[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:36 dist-test-slave-kzfr krb5kdc[14401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:36 dist-test-slave-kzfr krb5kdc[14401](info): set up 2 sockets
Sep 28 22:53:36 dist-test-slave-kzfr krb5kdc[14401](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 22:53:37.378089 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:53:37 dist-test-slave-kzfr krb5kdc[14401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:53:37.501777 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:45867
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:45867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:37.583107 14414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:37.583362 14414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:37.583420 14414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:37.584616 14414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:37.584661 14414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:37.584676 14414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:37.584689 14414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:37.584702 14414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:37.586229 14414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:45867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:45867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:37.586504 14414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:37.586759 14414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:37.592298 14422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:37.592384 14414 server_base.cc:1034] running on GCE node
W20260928 22:53:37.592351 14419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:37.592351 14420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:37.592839 14414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:37.593125 14414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:37.594286 14414 hybrid_clock.cc:648] HybridClock initialized: now 1790636017594272 us; error 27 us; skew 500 ppm
Sep 28 22:53:37 dist-test-slave-kzfr krb5kdc[14401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636017, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:37.596756 14414 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:53:37.597244 14414 webserver.cc:460] Webserver started at http://127.14.14.190:38491/ using document root <none> and password file <none>
I20260928 22:53:37.597524 14414 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:37.597615 14414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:37.597775 14414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:37.598938 14414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "9562d5606bad4491999b56196861ae13"
format_stamp: "Formatted at 2026-09-28 22:53:37 on dist-test-slave-kzfr"
server_key: "21b0ee2f59beab9b8d73983cb880580e"
server_key_iv: "575ac01c9824d73260a12b40d43201f5"
server_key_version: "encryptionkey@0"
I20260928 22:53:37.599319 14414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "9562d5606bad4491999b56196861ae13"
format_stamp: "Formatted at 2026-09-28 22:53:37 on dist-test-slave-kzfr"
server_key: "21b0ee2f59beab9b8d73983cb880580e"
server_key_iv: "575ac01c9824d73260a12b40d43201f5"
server_key_version: "encryptionkey@0"
I20260928 22:53:37.600775 14414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:37.601497 14429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:37.601684 14414 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:37.601780 14414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "9562d5606bad4491999b56196861ae13"
format_stamp: "Formatted at 2026-09-28 22:53:37 on dist-test-slave-kzfr"
server_key: "21b0ee2f59beab9b8d73983cb880580e"
server_key_iv: "575ac01c9824d73260a12b40d43201f5"
server_key_version: "encryptionkey@0"
I20260928 22:53:37.601864 14414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:37.620368 14414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:37.622239 14414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:37.622403 14414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:37.626359 14414 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:45867
I20260928 22:53:37.626417 14481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:45867 every 8 connection(s)
I20260928 22:53:37.626784 14414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:37.627683 14482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:37.629976 14482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13: Bootstrap starting.
I20260928 22:53:37.630541 14482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:37.630775 14482 log.cc:824] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:37.631439 14482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13: No bootstrap required, opened a new log
I20260928 22:53:37.632980 14482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9562d5606bad4491999b56196861ae13" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 45867 } }
I20260928 22:53:37.633152 14482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:37.633177 14482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9562d5606bad4491999b56196861ae13, State: Initialized, Role: FOLLOWER
I20260928 22:53:37.633308 14482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9562d5606bad4491999b56196861ae13" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 45867 } }
I20260928 22:53:37.633385 14482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:37.633409 14482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:37.633438 14482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:37.633975 14482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9562d5606bad4491999b56196861ae13" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 45867 } }
I20260928 22:53:37.634069 14482 leader_election.cc:302] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9562d5606bad4491999b56196861ae13; no voters:
I20260928 22:53:37.634181 14482 leader_election.cc:288] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:37.634275 14485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:37.634425 14482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:37.634590 14485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [term 1 LEADER]: Becoming Leader. State: Replica: 9562d5606bad4491999b56196861ae13, State: Running, Role: LEADER
I20260928 22:53:37.634692 14485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9562d5606bad4491999b56196861ae13" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 45867 } }
I20260928 22:53:37.635103 14486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9562d5606bad4491999b56196861ae13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9562d5606bad4491999b56196861ae13" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 45867 } } }
I20260928 22:53:37.635191 14486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:37.635244 14487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9562d5606bad4491999b56196861ae13. Latest consensus state: current_term: 1 leader_uuid: "9562d5606bad4491999b56196861ae13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9562d5606bad4491999b56196861ae13" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 45867 } } }
I20260928 22:53:37.635340 14487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:37.635510 14494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:37.636108 14494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:37.636562 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 14414
I20260928 22:53:37.636689 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:53:37.637336 14394 external_mini_cluster.cc:1428] Setting key 0b9ac405739481b1a759b21692aa7224
I20260928 22:53:37.637979 14494 catalog_manager.cc:1347] Generated new cluster ID: 0d059e42977c4ad795b7a28d114e2d45
I20260928 22:53:37.638062 14494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:37.645083 14494 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:37.645764 14494 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 22:53:37 dist-test-slave-kzfr krb5kdc[14401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:37.654858 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:37.638371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.14.14.190:45867)
0928 22:53:37.638548 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:37.638558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:37.638812 (+ 254us) server_negotiation.cc:407] Connection header received
0928 22:53:37.640416 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:37.640423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:37.640499 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:37.640652 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:37.642160 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:37.643668 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:37.644657 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:37.645300 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:37.645423 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:37.649930 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:37.649934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:37.649935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:37.649948 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:37.652662 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:37.653245 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:37.653247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:37.653248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:37.653349 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:37.653709 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:37.653710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:37.653710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:37.653988 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:37.654144 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:37.654490 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 22:53:37.654638 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260928 22:53:37.658355 14494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9562d5606bad4491999b56196861ae13: Generated new TSK 0
I20260928 22:53:37.658558 14494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:53:37.688905 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:45867
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:37.770779 14508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:37.770998 14508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:37.771023 14508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:37.772207 14508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:37.772253 14508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:37.772280 14508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:53:37.773880 14508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:45867
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:37.774151 14508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:37.774391 14508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:37.779758 14514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:37.780085 14508 server_base.cc:1034] running on GCE node
W20260928 22:53:37.779836 14516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:37.779758 14513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:37.780380 14508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:37.780668 14508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:37.781827 14508 hybrid_clock.cc:648] HybridClock initialized: now 1790636017781807 us; error 38 us; skew 500 ppm
Sep 28 22:53:37 dist-test-slave-kzfr krb5kdc[14401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636017, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:37.784596 14508 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:53:37.785104 14508 webserver.cc:460] Webserver started at http://127.14.14.129:42391/ using document root <none> and password file <none>
I20260928 22:53:37.785338 14508 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:37.785408 14508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:37.785559 14508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:37.786646 14508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "bd7f145e866341a3992549c4810fb36f"
format_stamp: "Formatted at 2026-09-28 22:53:37 on dist-test-slave-kzfr"
server_key: "a893c02b69246570ea7589e3854307df"
server_key_iv: "b11625b353ee9dc240c74c70225dfb9d"
server_key_version: "encryptionkey@0"
I20260928 22:53:37.787024 14508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "bd7f145e866341a3992549c4810fb36f"
format_stamp: "Formatted at 2026-09-28 22:53:37 on dist-test-slave-kzfr"
server_key: "a893c02b69246570ea7589e3854307df"
server_key_iv: "b11625b353ee9dc240c74c70225dfb9d"
server_key_version: "encryptionkey@0"
I20260928 22:53:37.788497 14508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:53:37.789314 14523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:37.789701 14508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:53:37.789804 14508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "bd7f145e866341a3992549c4810fb36f"
format_stamp: "Formatted at 2026-09-28 22:53:37 on dist-test-slave-kzfr"
server_key: "a893c02b69246570ea7589e3854307df"
server_key_iv: "b11625b353ee9dc240c74c70225dfb9d"
server_key_version: "encryptionkey@0"
I20260928 22:53:37.789889 14508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:37.804507 14508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:37.807494 14508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:37.807647 14508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:37.807898 14508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:37.808297 14508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:37.808364 14508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:37.808409 14508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:37.808431 14508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:37.815640 14508 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:38631
I20260928 22:53:37.815727 14636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:38631 every 8 connection(s)
I20260928 22:53:37.816045 14508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:53:37 dist-test-slave-kzfr krb5kdc[14401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636017, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:37.824055 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 14508
I20260928 22:53:37.824167 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:53:37.824337 14394 external_mini_cluster.cc:1428] Setting key 82b9ea01430e4f5ac05fa3c9af692df5
I20260928 22:53:37.826704 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:37.817061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:60343 (local address 127.14.14.190:45867)
0928 22:53:37.817116 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:37.817118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:37.817329 (+ 211us) server_negotiation.cc:407] Connection header received
0928 22:53:37.818110 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:37.818111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:37.818121 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:37.818172 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:37.819124 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:37.819618 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:37.820682 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:37.820874 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:37.820924 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:37.823184 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:37.823185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:37.823186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:37.823193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:37.824856 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:37.825438 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:37.825439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:37.825440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:37.825497 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:37.825824 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:37.825824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:37.825825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:37.825939 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:37.825995 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:37.826411 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 22:53:37.826543 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:53:37.826797 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:37.817018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:45867 (local address 127.14.14.129:60343)
0928 22:53:37.817193 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:37.817236 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:37.818010 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:37.818245 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:37.818249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:37.818839 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:37.819046 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:37.819051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:37.819706 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:37.819707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:37.820604 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:37.820606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:37.820944 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:37.820945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:37.821041 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:37.821655 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:37.821663 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:37.823083 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:37.824950 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:37.824957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:37.824960 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:37.825295 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:37.825571 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:37.825572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:37.825572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:37.825756 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:37.826007 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:37.826013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:37.826188 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 22:53:37.826300 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 22:53:37.826484 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260928 22:53:37.827271 14637 heartbeater.cc:347] Connected to a master server at 127.14.14.190:45867
I20260928 22:53:37.827379 14637 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:37.827600 14637 heartbeater.cc:511] Master 127.14.14.190:45867 requested a full tablet report, sending...
I20260928 22:53:37.828115 14446 ts_manager.cc:194] Registered new tserver with Master: bd7f145e866341a3992549c4810fb36f (127.14.14.129:38631)
I20260928 22:53:37.828809 14446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:60343
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:53:37.852921 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:45867
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:37.933874 14642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:37.934073 14642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:37.934096 14642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:37.935360 14642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:37.935402 14642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:37.935429 14642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:53:37.936921 14642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:45867
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:37.937199 14642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:37.937433 14642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:53:37.942461 14642 server_base.cc:1034] running on GCE node
W20260928 22:53:37.942445 14647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:37.942435 14650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:37.942445 14648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:37.942839 14642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:37.943029 14642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:37.944176 14642 hybrid_clock.cc:648] HybridClock initialized: now 1790636017944157 us; error 36 us; skew 500 ppm
Sep 28 22:53:37 dist-test-slave-kzfr krb5kdc[14401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636017, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:37.946661 14642 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:53:37.947274 14642 webserver.cc:460] Webserver started at http://127.14.14.130:40477/ using document root <none> and password file <none>
I20260928 22:53:37.947548 14642 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:37.947597 14642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:37.947767 14642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:37.948906 14642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "5f9cfa1f3b204f9b94ab038749a9144f"
format_stamp: "Formatted at 2026-09-28 22:53:37 on dist-test-slave-kzfr"
server_key: "23f9bc0bbe2a7ebbe708fc4115ab8f18"
server_key_iv: "2992a15239fe0c4333d830d0de09c3d6"
server_key_version: "encryptionkey@0"
I20260928 22:53:37.949265 14642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "5f9cfa1f3b204f9b94ab038749a9144f"
format_stamp: "Formatted at 2026-09-28 22:53:37 on dist-test-slave-kzfr"
server_key: "23f9bc0bbe2a7ebbe708fc4115ab8f18"
server_key_iv: "2992a15239fe0c4333d830d0de09c3d6"
server_key_version: "encryptionkey@0"
I20260928 22:53:37.950686 14642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:53:37.951565 14657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:37.951778 14642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:53:37.951879 14642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "5f9cfa1f3b204f9b94ab038749a9144f"
format_stamp: "Formatted at 2026-09-28 22:53:37 on dist-test-slave-kzfr"
server_key: "23f9bc0bbe2a7ebbe708fc4115ab8f18"
server_key_iv: "2992a15239fe0c4333d830d0de09c3d6"
server_key_version: "encryptionkey@0"
I20260928 22:53:37.951957 14642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:37.976199 14642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:37.978464 14642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:37.978662 14642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:37.978914 14642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:37.979251 14642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:37.979312 14642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:37.979355 14642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:37.979384 14642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:37.985637 14642 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:33583
I20260928 22:53:37.985711 14770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:33583 every 8 connection(s)
I20260928 22:53:37.986073 14642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:53:37.987769 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 14642
I20260928 22:53:37.987882 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:53:37.988054 14394 external_mini_cluster.cc:1428] Setting key 09d3962194005491cd22d66b3f81a532
Sep 28 22:53:37 dist-test-slave-kzfr krb5kdc[14401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636017, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
I20260928 22:53:37.998247 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:37.986961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:46135 (local address 127.14.14.190:45867)
0928 22:53:37.987053 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:37.987055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:37.987265 (+ 210us) server_negotiation.cc:407] Connection header received
0928 22:53:37.987917 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:37.987918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:37.987933 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:37.987998 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:37.989257 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:37.990249 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:37.991861 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:37.992069 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:37.992133 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:37.994860 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:37.994861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:37.994862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:37.994871 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:37.996691 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:37.997193 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:37.997195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:37.997195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:37.997261 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:37.997529 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:37.997530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:37.997531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:37.997714 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:37.997793 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:37.998067 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 22:53:37.998152 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 22:53:37.998485 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:37.986933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:45867 (local address 127.14.14.130:46135)
0928 22:53:37.987138 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:37.987175 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:37.987789 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:37.988102 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:37.988107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:37.988911 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:37.989146 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:37.989153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:37.990355 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:37.990357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:37.991763 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:37.991766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:37.992159 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:37.992160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:37.992313 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:37.992901 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:37.992913 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:37.994760 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:37.996797 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:37.996802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:37.996804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:37.997084 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:37.997335 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:37.997336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:37.997336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:37.997449 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:37.997783 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:37.997790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:37.997930 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 22:53:37.998046 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 22:53:37.998211 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":113,"threads_started":1}
I20260928 22:53:37.999045 14771 heartbeater.cc:347] Connected to a master server at 127.14.14.190:45867
I20260928 22:53:37.999151 14771 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:37.999341 14771 heartbeater.cc:511] Master 127.14.14.190:45867 requested a full tablet report, sending...
I20260928 22:53:37.999805 14446 ts_manager.cc:194] Registered new tserver with Master: 5f9cfa1f3b204f9b94ab038749a9144f (127.14.14.130:33583)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
I20260928 22:53:38.000207 14446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:46135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:53:38.016710 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:45867
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:38.099313 14776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:38.099519 14776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:38.099543 14776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:38.100690 14776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:38.100733 14776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:38.100785 14776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:53:38.102428 14776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:45867
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:38.102679 14776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:38.102914 14776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:38.108166 14782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:38.108210 14776 server_base.cc:1034] running on GCE node
W20260928 22:53:38.108181 14781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:38.108171 14784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:38.108587 14776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:38.108777 14776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:38.109910 14776 hybrid_clock.cc:648] HybridClock initialized: now 1790636018109899 us; error 28 us; skew 500 ppm
Sep 28 22:53:38 dist-test-slave-kzfr krb5kdc[14401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636018, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:38.112581 14776 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:53:38.113008 14776 webserver.cc:460] Webserver started at http://127.14.14.131:43681/ using document root <none> and password file <none>
I20260928 22:53:38.113251 14776 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:38.113319 14776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:38.113469 14776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:38.114530 14776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "465984aafba7462fbc31fc072b272d7e"
format_stamp: "Formatted at 2026-09-28 22:53:38 on dist-test-slave-kzfr"
server_key: "d4c71ffde0a343877d099b8c415f99ed"
server_key_iv: "34bf5176e5e33e00d69e98c513282354"
server_key_version: "encryptionkey@0"
I20260928 22:53:38.114884 14776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "465984aafba7462fbc31fc072b272d7e"
format_stamp: "Formatted at 2026-09-28 22:53:38 on dist-test-slave-kzfr"
server_key: "d4c71ffde0a343877d099b8c415f99ed"
server_key_iv: "34bf5176e5e33e00d69e98c513282354"
server_key_version: "encryptionkey@0"
I20260928 22:53:38.116170 14776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:53:38.116873 14791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:38.117045 14776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:38.117148 14776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "465984aafba7462fbc31fc072b272d7e"
format_stamp: "Formatted at 2026-09-28 22:53:38 on dist-test-slave-kzfr"
server_key: "d4c71ffde0a343877d099b8c415f99ed"
server_key_iv: "34bf5176e5e33e00d69e98c513282354"
server_key_version: "encryptionkey@0"
I20260928 22:53:38.117234 14776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:38.127966 14776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:38.130225 14776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:38.130383 14776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:38.130604 14776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:38.130898 14776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:38.130954 14776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:38.130996 14776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:38.131027 14776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:38.137702 14776 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:33559
I20260928 22:53:38.137743 14904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:33559 every 8 connection(s)
I20260928 22:53:38.138073 14776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:53:38.141228 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 14776
I20260928 22:53:38.141309 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:53:38.141489 14394 external_mini_cluster.cc:1428] Setting key feed35d7ca8969ad5723b1a66b75b3c7
Sep 28 22:53:38 dist-test-slave-kzfr krb5kdc[14401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636018, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:38.147197 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:38.138920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:42281 (local address 127.14.14.190:45867)
0928 22:53:38.138980 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:38.138982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:38.139005 (+ 23us) server_negotiation.cc:407] Connection header received
0928 22:53:38.139481 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:38.139482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:38.139491 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:38.139556 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:38.140437 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:38.140920 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:38.142171 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:38.142377 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:38.142464 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:38.144538 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:38.144540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:38.144541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:38.144548 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:38.145856 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:38.146253 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:38.146254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:38.146255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:38.146314 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:38.146567 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:38.146568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:38.146569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:38.146680 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:38.146757 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:38.147018 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 22:53:38.147103 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 22:53:38.147411 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:38.138738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:45867 (local address 127.14.14.131:42281)
0928 22:53:38.138892 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:38.138934 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:38.139399 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:38.139611 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:38.139614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:38.140156 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:38.140358 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:38.140363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:38.140998 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:38.140998 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:38.142073 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:38.142075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:38.142472 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:38.142474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:38.142613 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:38.143133 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:38.143141 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:38.144442 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:38.145948 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:38.145954 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:38.145957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:38.146176 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:38.146382 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:38.146383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:38.146384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:38.146492 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:38.146748 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:38.146754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:38.146889 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 22:53:38.146998 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 22:53:38.147184 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260928 22:53:38.147902 14905 heartbeater.cc:347] Connected to a master server at 127.14.14.190:45867
I20260928 22:53:38.147984 14905 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:38.148216 14905 heartbeater.cc:511] Master 127.14.14.190:45867 requested a full tablet report, sending...
I20260928 22:53:38.148571 14446 ts_manager.cc:194] Registered new tserver with Master: 465984aafba7462fbc31fc072b272d7e (127.14.14.131:33559)
I20260928 22:53:38.148947 14446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:42281
I20260928 22:53:38.153689 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 22:53:38 dist-test-slave-kzfr krb5kdc[14401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636018, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 22:53:38 dist-test-slave-kzfr krb5kdc[14401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636018, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.14.129@KRBTEST.COM
I20260928 22:53:38.176929 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:38.169022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.14.14.129:38631)
0928 22:53:38.169192 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:38.169196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:38.169205 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:53:38.169230 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:38.169232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:38.169245 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:38.169332 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:38.170003 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:38.170752 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:38.171500 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:38.171831 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:38.171928 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:38.174361 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:38.174369 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:38.174369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:38.174376 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:38.175826 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:38.176179 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:38.176183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:38.176183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:38.176250 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:38.176441 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:38.176442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:38.176443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:38.176605 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:38.176731 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:38.176788 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 22:53:38.176850 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
W20260928 22:53:38.177371 14551 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36566
Sep 28 22:53:38 dist-test-slave-kzfr krb5kdc[14401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 22:53:38 dist-test-slave-kzfr krb5kdc[14401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.129@KRBTEST.COM
I20260928 22:53:38.200169 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:38.192974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36576 (local address 127.14.14.129:38631)
0928 22:53:38.193053 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:38.193055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:38.193161 (+ 106us) server_negotiation.cc:407] Connection header received
0928 22:53:38.193227 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:38.193228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:38.193238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:38.193313 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:38.193958 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:38.194444 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:38.195143 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:38.195334 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:38.195384 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:38.197569 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:38.197570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:38.197570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:38.197606 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:38.199021 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:38.199388 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:38.199389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:38.199390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:38.199448 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:38.199653 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:38.199654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:38.199654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:38.199759 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:38.199882 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:38.199974 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 22:53:38.200060 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:53:38.200775 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 14508
I20260928 22:53:38.207095 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 14642
I20260928 22:53:38.211803 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 14776
I20260928 22:53:38.216143 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 14414
2026-09-28T22:53:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1969 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:38 dist-test-slave-kzfr krb5kdc[14919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:38 dist-test-slave-kzfr krb5kdc[14919](info): set up 2 sockets
Sep 28 22:53:38 dist-test-slave-kzfr krb5kdc[14919](info): commencing operation
krb5kdc: starting...
W20260928 22:53:39.273530 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:53:39 dist-test-slave-kzfr krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:53:39.392835 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:36183
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:36183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:39.477305 14932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:39.477576 14932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:39.477631 14932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:39.478801 14932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:39.478879 14932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:39.478904 14932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:39.478925 14932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:39.478952 14932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:39.480502 14932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:36183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:36183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:39.480796 14932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:39.481033 14932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:39.486127 14938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:39.486135 14937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:39.486167 14932 server_base.cc:1034] running on GCE node
W20260928 22:53:39.486155 14940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:39.486670 14932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:39.486971 14932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:39.488116 14932 hybrid_clock.cc:648] HybridClock initialized: now 1790636019488089 us; error 43 us; skew 500 ppm
Sep 28 22:53:39 dist-test-slave-kzfr krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636019, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:39.490593 14932 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:53:39.491088 14932 webserver.cc:460] Webserver started at http://127.14.14.190:35791/ using document root <none> and password file <none>
I20260928 22:53:39.491319 14932 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:39.491386 14932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:39.491540 14932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:39.492627 14932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "81ba1a8cb0724a2c9ce7e33b2ddffabe"
format_stamp: "Formatted at 2026-09-28 22:53:39 on dist-test-slave-kzfr"
server_key: "63cefe71d61cac3e0fee4b5829404b0a"
server_key_iv: "2708ebc29d2dc5f2762aaf32b33ee852"
server_key_version: "encryptionkey@0"
I20260928 22:53:39.493024 14932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "81ba1a8cb0724a2c9ce7e33b2ddffabe"
format_stamp: "Formatted at 2026-09-28 22:53:39 on dist-test-slave-kzfr"
server_key: "63cefe71d61cac3e0fee4b5829404b0a"
server_key_iv: "2708ebc29d2dc5f2762aaf32b33ee852"
server_key_version: "encryptionkey@0"
I20260928 22:53:39.494689 14932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:53:39.495458 14947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:39.495760 14932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:53:39.495867 14932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "81ba1a8cb0724a2c9ce7e33b2ddffabe"
format_stamp: "Formatted at 2026-09-28 22:53:39 on dist-test-slave-kzfr"
server_key: "63cefe71d61cac3e0fee4b5829404b0a"
server_key_iv: "2708ebc29d2dc5f2762aaf32b33ee852"
server_key_version: "encryptionkey@0"
I20260928 22:53:39.495940 14932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:39.517306 14932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:39.519198 14932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:39.519362 14932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:39.523141 14932 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:36183
I20260928 22:53:39.523175 14999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:36183 every 8 connection(s)
I20260928 22:53:39.523532 14932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:39.524385 15000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:39.526626 15000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe: Bootstrap starting.
I20260928 22:53:39.527175 15000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:39.527405 15000 log.cc:824] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:39.527873 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 14932
I20260928 22:53:39.527971 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:53:39.528122 14394 external_mini_cluster.cc:1428] Setting key 49e4d45bfc36861425c46172036a6120
I20260928 22:53:39.528124 15000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe: No bootstrap required, opened a new log
I20260928 22:53:39.530395 15000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ba1a8cb0724a2c9ce7e33b2ddffabe" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36183 } }
I20260928 22:53:39.530586 15000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:39.530630 15000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ba1a8cb0724a2c9ce7e33b2ddffabe, State: Initialized, Role: FOLLOWER
I20260928 22:53:39.530779 15000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ba1a8cb0724a2c9ce7e33b2ddffabe" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36183 } }
I20260928 22:53:39.530854 15000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:39.530884 15000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:39.530928 15000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:39.531572 15000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ba1a8cb0724a2c9ce7e33b2ddffabe" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36183 } }
I20260928 22:53:39.531675 15000 leader_election.cc:302] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81ba1a8cb0724a2c9ce7e33b2ddffabe; no voters:
I20260928 22:53:39.531821 15000 leader_election.cc:288] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:39.531895 15005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:39.532029 15005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [term 1 LEADER]: Becoming Leader. State: Replica: 81ba1a8cb0724a2c9ce7e33b2ddffabe, State: Running, Role: LEADER
I20260928 22:53:39.532088 15000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:39.532150 15005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ba1a8cb0724a2c9ce7e33b2ddffabe" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36183 } }
I20260928 22:53:39.532418 15005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81ba1a8cb0724a2c9ce7e33b2ddffabe. Latest consensus state: current_term: 1 leader_uuid: "81ba1a8cb0724a2c9ce7e33b2ddffabe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ba1a8cb0724a2c9ce7e33b2ddffabe" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36183 } } }
I20260928 22:53:39.532505 15005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:39.532429 15007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81ba1a8cb0724a2c9ce7e33b2ddffabe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ba1a8cb0724a2c9ce7e33b2ddffabe" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36183 } } }
I20260928 22:53:39.532652 15007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:39.532771 15010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:39.533289 15010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:39.535012 15010 catalog_manager.cc:1347] Generated new cluster ID: 34a6a5d1da9f4ca7966b380d96ed3b1c
I20260928 22:53:39.535081 15010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:53:39 dist-test-slave-kzfr krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:39.540906 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:39.528657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33780 (local address 127.14.14.190:36183)
0928 22:53:39.528847 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:39.528857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:39.528894 (+ 37us) server_negotiation.cc:407] Connection header received
0928 22:53:39.529578 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:39.529588 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:39.529670 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:39.529810 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:39.530358 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:39.532071 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:39.533265 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:39.533912 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:39.534058 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:39.536757 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:39.536760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:39.536761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:39.536771 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:39.538898 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:39.539290 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:39.539293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:39.539294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:39.539390 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:39.539613 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:39.539613 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:39.539614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:39.539903 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:39.540042 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:39.540564 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 22:53:39.540712 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260928 22:53:39.543720 15010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:39.544435 15010 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:39.549813 15010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81ba1a8cb0724a2c9ce7e33b2ddffabe: Generated new TSK 0
I20260928 22:53:39.549978 15010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:53:39.576659 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36183
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:39.656725 15026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:39.656975 15026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:39.657015 15026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:39.658200 15026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:39.658272 15026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:39.658318 15026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:53:39.659870 15026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:39.660156 15026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:39.660420 15026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:39.665454 15032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:39.665454 15034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:39.665455 15031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:39.665668 15026 server_base.cc:1034] running on GCE node
I20260928 22:53:39.665982 15026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:39.666237 15026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:39.667400 15026 hybrid_clock.cc:648] HybridClock initialized: now 1790636019667397 us; error 38 us; skew 500 ppm
Sep 28 22:53:39 dist-test-slave-kzfr krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636019, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:39.669866 15026 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:53:39.670349 15026 webserver.cc:460] Webserver started at http://127.14.14.129:46407/ using document root <none> and password file <none>
I20260928 22:53:39.670588 15026 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:39.670658 15026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:39.670809 15026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:39.671868 15026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "451c10fe90714adaa953f0322ee2b2c6"
format_stamp: "Formatted at 2026-09-28 22:53:39 on dist-test-slave-kzfr"
server_key: "a432ff457029b25912f11dfc58a35975"
server_key_iv: "86c8bac2e6c55d14505956f1c04c4822"
server_key_version: "encryptionkey@0"
I20260928 22:53:39.672235 15026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "451c10fe90714adaa953f0322ee2b2c6"
format_stamp: "Formatted at 2026-09-28 22:53:39 on dist-test-slave-kzfr"
server_key: "a432ff457029b25912f11dfc58a35975"
server_key_iv: "86c8bac2e6c55d14505956f1c04c4822"
server_key_version: "encryptionkey@0"
I20260928 22:53:39.673557 15026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 22:53:39.674399 15041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:39.674587 15026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:39.674683 15026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "451c10fe90714adaa953f0322ee2b2c6"
format_stamp: "Formatted at 2026-09-28 22:53:39 on dist-test-slave-kzfr"
server_key: "a432ff457029b25912f11dfc58a35975"
server_key_iv: "86c8bac2e6c55d14505956f1c04c4822"
server_key_version: "encryptionkey@0"
I20260928 22:53:39.674770 15026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:39.686031 15026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:39.687986 15026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:39.688150 15026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:39.688383 15026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:39.688699 15026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:39.688753 15026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:39.688794 15026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:39.688823 15026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:39.695017 15026 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:46159
I20260928 22:53:39.695057 15154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:46159 every 8 connection(s)
I20260928 22:53:39.695412 15026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:53:39.701411 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 15026
I20260928 22:53:39.701510 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:53:39.701702 14394 external_mini_cluster.cc:1428] Setting key 8e18d56f5a03987338db37d67289735f
Sep 28 22:53:39 dist-test-slave-kzfr krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636019, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:39.705265 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:39.696257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:46423 (local address 127.14.14.190:36183)
0928 22:53:39.696323 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:39.696326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:39.696542 (+ 216us) server_negotiation.cc:407] Connection header received
0928 22:53:39.697502 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:39.697504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:39.697514 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:39.697564 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:39.698225 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:39.698728 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:39.699872 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:39.700055 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:39.700109 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:39.702540 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:39.702541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:39.702541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:39.702549 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:39.703890 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:39.704309 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:39.704310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:39.704311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:39.704368 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:39.704622 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:39.704623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:39.704624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:39.704736 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:39.704808 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:39.705086 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 22:53:39.705173 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:53:39.705632 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:39.696237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:36183 (local address 127.14.14.129:46423)
0928 22:53:39.696420 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:39.696456 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:39.697344 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:39.697682 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:39.697686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:39.697947 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:39.698146 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:39.698150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:39.698814 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:39.698816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:39.699804 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:39.699806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:39.700126 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:39.700127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:39.700242 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:39.700859 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:39.700867 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:39.702432 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:39.703984 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:39.703990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:39.703992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:39.704220 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:39.704434 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:39.704435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:39.704436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:39.704549 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:39.704813 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:39.704820 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:39.704957 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 22:53:39.705065 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 22:53:39.705228 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260928 22:53:39.706159 15155 heartbeater.cc:347] Connected to a master server at 127.14.14.190:36183
I20260928 22:53:39.706264 15155 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:39.706465 15155 heartbeater.cc:511] Master 127.14.14.190:36183 requested a full tablet report, sending...
I20260928 22:53:39.706955 14964 ts_manager.cc:194] Registered new tserver with Master: 451c10fe90714adaa953f0322ee2b2c6 (127.14.14.129:46159)
I20260928 22:53:39.707580 14964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:46423
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:53:39.728358 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36183
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:39.812074 15160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:39.812306 15160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:39.812347 15160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:39.813488 15160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:39.813557 15160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:39.813619 15160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:53:39.815094 15160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:39.815364 15160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:39.815609 15160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:39.820856 15165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:39.820933 15160 server_base.cc:1034] running on GCE node
W20260928 22:53:39.820899 15168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:39.820899 15166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:39.821372 15160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:39.821734 15160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:39.822906 15160 hybrid_clock.cc:648] HybridClock initialized: now 1790636019822889 us; error 41 us; skew 500 ppm
Sep 28 22:53:39 dist-test-slave-kzfr krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636019, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:39.825645 15160 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:53:39.826166 15160 webserver.cc:460] Webserver started at http://127.14.14.130:39119/ using document root <none> and password file <none>
I20260928 22:53:39.826413 15160 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:39.826495 15160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:39.826644 15160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:39.827672 15160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "cc8b3a068ab24554a4eaf03174d60bca"
format_stamp: "Formatted at 2026-09-28 22:53:39 on dist-test-slave-kzfr"
server_key: "f1aea14ec314b98395b7bb6a86dc1cf4"
server_key_iv: "bb75914f31186cc7db1347adf5ea28ba"
server_key_version: "encryptionkey@0"
I20260928 22:53:39.828027 15160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "cc8b3a068ab24554a4eaf03174d60bca"
format_stamp: "Formatted at 2026-09-28 22:53:39 on dist-test-slave-kzfr"
server_key: "f1aea14ec314b98395b7bb6a86dc1cf4"
server_key_iv: "bb75914f31186cc7db1347adf5ea28ba"
server_key_version: "encryptionkey@0"
I20260928 22:53:39.829308 15160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:39.830133 15175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:39.830291 15160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:39.830385 15160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "cc8b3a068ab24554a4eaf03174d60bca"
format_stamp: "Formatted at 2026-09-28 22:53:39 on dist-test-slave-kzfr"
server_key: "f1aea14ec314b98395b7bb6a86dc1cf4"
server_key_iv: "bb75914f31186cc7db1347adf5ea28ba"
server_key_version: "encryptionkey@0"
I20260928 22:53:39.830474 15160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:39.843883 15160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:39.846218 15160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:39.846383 15160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:39.846613 15160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:39.846927 15160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:39.846984 15160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:39.847023 15160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:39.847049 15160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:39.852882 15160 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:36181
I20260928 22:53:39.852991 15288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:36181 every 8 connection(s)
I20260928 22:53:39.853274 15160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:53:39 dist-test-slave-kzfr krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636019, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:39.862946 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:39.853916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:48389 (local address 127.14.14.190:36183)
0928 22:53:39.853999 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:39.854002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:39.854258 (+ 256us) server_negotiation.cc:407] Connection header received
0928 22:53:39.855243 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:39.855244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:39.855255 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:39.855307 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:39.855943 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:39.856432 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:39.857450 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:39.857697 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:39.857744 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:39.860193 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:39.860194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:39.860195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:39.860203 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:39.861366 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:39.861979 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:39.861980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:39.861980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:39.862028 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:39.862287 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:39.862288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:39.862289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:39.862401 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:39.862476 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:39.862749 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 22:53:39.862827 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 22:53:39.863127 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:39.853911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:36183 (local address 127.14.14.130:48389)
0928 22:53:39.854133 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:39.854169 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:39.855151 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:39.855362 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:39.855365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:39.855626 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:39.855849 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:39.855853 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:39.856539 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:39.856540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:39.857367 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:39.857370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:39.857764 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:39.857765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:39.857874 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:39.858448 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:39.858457 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:39.860092 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:39.861461 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:39.861467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:39.861470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:39.861764 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:39.862102 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:39.862103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:39.862103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:39.862218 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:39.862468 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:39.862473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:39.862615 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 22:53:39.862725 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 22:53:39.862885 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":111,"threads_started":1}
I20260928 22:53:39.863147 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 15160
I20260928 22:53:39.863222 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:53:39.863384 14394 external_mini_cluster.cc:1428] Setting key db848b64e93e93a9bf9d9140acf636de
I20260928 22:53:39.863638 15289 heartbeater.cc:347] Connected to a master server at 127.14.14.190:36183
I20260928 22:53:39.863745 15289 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:39.863947 15289 heartbeater.cc:511] Master 127.14.14.190:36183 requested a full tablet report, sending...
I20260928 22:53:39.864368 14964 ts_manager.cc:194] Registered new tserver with Master: cc8b3a068ab24554a4eaf03174d60bca (127.14.14.130:36181)
I20260928 22:53:39.864858 14964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:48389
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:53:39.889262 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36183
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:39.968724 15294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:39.968983 15294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:39.969022 15294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:39.970188 15294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:39.970263 15294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:39.970309 15294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:53:39.971791 15294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:39.972057 15294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:39.972294 15294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:39.977362 15302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:39.977314 15300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:39.977483 15294 server_base.cc:1034] running on GCE node
W20260928 22:53:39.977314 15299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:39.977852 15294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:39.978133 15294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:39.979321 15294 hybrid_clock.cc:648] HybridClock initialized: now 1790636019979315 us; error 36 us; skew 500 ppm
Sep 28 22:53:39 dist-test-slave-kzfr krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636019, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:39.981846 15294 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:53:39.982364 15294 webserver.cc:460] Webserver started at http://127.14.14.131:45403/ using document root <none> and password file <none>
I20260928 22:53:39.982585 15294 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:39.982627 15294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:39.982730 15294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:39.983716 15294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "0aaef46e02254a468bdbafbe8a4541a4"
format_stamp: "Formatted at 2026-09-28 22:53:39 on dist-test-slave-kzfr"
server_key: "ab63ccb4c326a92084580ce5ee7c1a21"
server_key_iv: "adf61065c25e95319c053a0794e8e2cc"
server_key_version: "encryptionkey@0"
I20260928 22:53:39.984012 15294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "0aaef46e02254a468bdbafbe8a4541a4"
format_stamp: "Formatted at 2026-09-28 22:53:39 on dist-test-slave-kzfr"
server_key: "ab63ccb4c326a92084580ce5ee7c1a21"
server_key_iv: "adf61065c25e95319c053a0794e8e2cc"
server_key_version: "encryptionkey@0"
I20260928 22:53:39.985246 15294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:53:39.986030 15309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:39.986246 15294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:53:39.986313 15294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "0aaef46e02254a468bdbafbe8a4541a4"
format_stamp: "Formatted at 2026-09-28 22:53:39 on dist-test-slave-kzfr"
server_key: "ab63ccb4c326a92084580ce5ee7c1a21"
server_key_iv: "adf61065c25e95319c053a0794e8e2cc"
server_key_version: "encryptionkey@0"
I20260928 22:53:39.986363 15294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:40.022509 15294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:40.024925 15294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:40.025038 15294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:40.025218 15294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:40.025538 15294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:40.025622 15294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:40.025689 15294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:40.025720 15294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:40.031790 15294 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:45867
I20260928 22:53:40.031850 15422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:45867 every 8 connection(s)
I20260928 22:53:40.032193 15294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:53:40.034296 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 15294
I20260928 22:53:40.034432 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:53:40.034642 14394 external_mini_cluster.cc:1428] Setting key 8149e69ee90c830aae7226cfc456300b
Sep 28 22:53:40 dist-test-slave-kzfr krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636019, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:40.042084 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.033039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:43523 (local address 127.14.14.190:36183)
0928 22:53:40.033080 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:40.033083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:40.033316 (+ 233us) server_negotiation.cc:407] Connection header received
0928 22:53:40.034581 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:40.034583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:40.034595 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:40.034646 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:40.035471 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.035985 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.037049 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.037248 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.037305 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.039511 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:40.039512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:40.039512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:40.039519 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:40.040785 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:40.041160 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:40.041161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:40.041161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:40.041209 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:40.041510 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:40.041511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:40.041511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:40.041623 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:40.041668 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:40.041952 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 22:53:40.042021 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:53:40.042152 15425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.033014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:36183 (local address 127.14.14.131:43523)
0928 22:53:40.033174 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:40.033220 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:40.034450 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:40.034713 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:40.034717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:40.035103 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:40.035382 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:40.035389 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.036064 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.036065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:40.036969 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:40.036971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.037333 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.037334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:40.037454 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.038006 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:40.038014 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:40.039432 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:40.040865 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:40.040870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:40.040872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:40.041096 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:40.041298 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:40.041299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:40.041299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:40.041444 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:40.041674 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:40.041679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:40.041790 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 22:53:40.041870 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 22:53:40.042000 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260928 22:53:40.042773 15423 heartbeater.cc:347] Connected to a master server at 127.14.14.190:36183
I20260928 22:53:40.042870 15423 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:40.043077 15423 heartbeater.cc:511] Master 127.14.14.190:36183 requested a full tablet report, sending...
I20260928 22:53:40.043519 14964 ts_manager.cc:194] Registered new tserver with Master: 0aaef46e02254a468bdbafbe8a4541a4 (127.14.14.131:45867)
I20260928 22:53:40.043898 14964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:43523
I20260928 22:53:40.046516 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:53:40.053455 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.048106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33782 (local address 127.14.14.190:36183)
0928 22:53:40.048163 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:40.048165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:40.048265 (+ 100us) server_negotiation.cc:407] Connection header received
0928 22:53:40.048314 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:40.048315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:40.048324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:40.048371 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:40.048884 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.049347 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.050100 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.050278 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.050334 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.051371 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:40.051372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:40.051373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:40.051378 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:40.052503 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:40.052834 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:40.052835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:40.052836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:40.052885 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:40.053046 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:40.053046 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:40.053047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:40.053179 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:40.053255 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:40.053340 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 22:53:40.053395 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:53:40.055572 14964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:53:40.057227 14964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:53:40.063758 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.059916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:36181 (local address 127.0.0.1:37552)
0928 22:53:40.060135 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:40.060146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:40.060199 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:40.060448 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:40.060452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:40.060455 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:40.060525 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:40.060527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.061542 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.061543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:40.062544 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:40.062546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.063459 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.063460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:40.063532 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.063546 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:53:40.063622 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 22:53:40.063679 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":72,"threads_started":1}
I20260928 22:53:40.063772 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.060117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.14.14.130:36181)
0928 22:53:40.060277 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:40.060281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:40.060289 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:53:40.060320 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:40.060321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:40.060334 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:40.060436 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:40.060586 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.061434 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.062652 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.063367 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.063448 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.063511 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:40.063647 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 22:53:40.063693 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260928 22:53:40.064282 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.059706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:46159 (local address 127.0.0.1:36184)
0928 22:53:40.059980 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:40.059991 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:40.060042 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:40.060713 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:40.060714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:40.060716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:40.060749 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:40.060750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.062175 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.062176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:40.063089 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:40.063091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.064043 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.064044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:40.064117 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.064121 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:40.064178 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:53:40.064215 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":4,"thread_start_us":67,"threads_started":1}
I20260928 22:53:40.064302 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.059824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.14.14.129:46159)
0928 22:53:40.060330 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:40.060333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:40.060341 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:53:40.060406 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:40.060408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:40.060426 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:40.060584 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:40.060831 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.061951 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.063167 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.063769 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.064024 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.064083 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:40.064190 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 22:53:40.064228 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":84,"threads_started":1}
I20260928 22:53:40.064469 15223 tablet_service.cc:1467] Processing CreateTablet for tablet 0ace26e572f445ea93fce6a31f0b98f1 (DEFAULT_TABLE table=test-table [id=8a73be86cdc04bae88ff12a7514017b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:40.064872 15223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ace26e572f445ea93fce6a31f0b98f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:40.064934 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.059706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:45867 (local address 127.0.0.1:40612)
0928 22:53:40.059926 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:40.059968 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:40.060042 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:40.061065 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:40.061066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:40.061068 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:40.061108 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:40.061109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.062329 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.062330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:40.063400 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:40.063403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.064719 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.064720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:40.064781 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.064785 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:40.064815 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 22:53:40.064857 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":62,"threads_started":1}
I20260928 22:53:40.065140 15089 tablet_service.cc:1467] Processing CreateTablet for tablet 0ace26e572f445ea93fce6a31f0b98f1 (DEFAULT_TABLE table=test-table [id=8a73be86cdc04bae88ff12a7514017b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:40.065320 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.059900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40612 (local address 127.14.14.131:45867)
0928 22:53:40.060906 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:40.060910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:40.060919 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:53:40.060954 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:40.060956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:40.060970 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:40.061047 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:40.061172 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.062232 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.063492 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.064616 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.065098 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.065158 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:40.065183 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 22:53:40.065236 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":973,"thread_start_us":89,"threads_started":1}
I20260928 22:53:40.065404 15089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ace26e572f445ea93fce6a31f0b98f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:40.066085 15357 tablet_service.cc:1467] Processing CreateTablet for tablet 0ace26e572f445ea93fce6a31f0b98f1 (DEFAULT_TABLE table=test-table [id=8a73be86cdc04bae88ff12a7514017b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:40.066421 15357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ace26e572f445ea93fce6a31f0b98f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:40.068354 15440 tablet_bootstrap.cc:492] T 0ace26e572f445ea93fce6a31f0b98f1 P 451c10fe90714adaa953f0322ee2b2c6: Bootstrap starting.
I20260928 22:53:40.068765 15441 tablet_bootstrap.cc:492] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca: Bootstrap starting.
I20260928 22:53:40.069017 15440 tablet_bootstrap.cc:654] T 0ace26e572f445ea93fce6a31f0b98f1 P 451c10fe90714adaa953f0322ee2b2c6: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:40.069391 15441 tablet_bootstrap.cc:654] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:40.069573 15440 log.cc:824] T 0ace26e572f445ea93fce6a31f0b98f1 P 451c10fe90714adaa953f0322ee2b2c6: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:40.069751 15442 tablet_bootstrap.cc:492] T 0ace26e572f445ea93fce6a31f0b98f1 P 0aaef46e02254a468bdbafbe8a4541a4: Bootstrap starting.
I20260928 22:53:40.070019 15441 log.cc:824] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:40.070415 15442 tablet_bootstrap.cc:654] T 0ace26e572f445ea93fce6a31f0b98f1 P 0aaef46e02254a468bdbafbe8a4541a4: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:40.070425 15440 tablet_bootstrap.cc:492] T 0ace26e572f445ea93fce6a31f0b98f1 P 451c10fe90714adaa953f0322ee2b2c6: No bootstrap required, opened a new log
I20260928 22:53:40.070485 15440 ts_tablet_manager.cc:1397] T 0ace26e572f445ea93fce6a31f0b98f1 P 451c10fe90714adaa953f0322ee2b2c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:53:40.070708 15441 tablet_bootstrap.cc:492] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca: No bootstrap required, opened a new log
I20260928 22:53:40.070791 15441 ts_tablet_manager.cc:1397] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:53:40.071033 15442 log.cc:824] T 0ace26e572f445ea93fce6a31f0b98f1 P 0aaef46e02254a468bdbafbe8a4541a4: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:40.072250 15440 raft_consensus.cc:348] T 0ace26e572f445ea93fce6a31f0b98f1 P 451c10fe90714adaa953f0322ee2b2c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c10fe90714adaa953f0322ee2b2c6" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46159 } } peers { permanent_uuid: "0aaef46e02254a468bdbafbe8a4541a4" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 45867 } } peers { permanent_uuid: "cc8b3a068ab24554a4eaf03174d60bca" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 36181 } }
I20260928 22:53:40.072392 15440 raft_consensus.cc:374] T 0ace26e572f445ea93fce6a31f0b98f1 P 451c10fe90714adaa953f0322ee2b2c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:40.072415 15440 raft_consensus.cc:729] T 0ace26e572f445ea93fce6a31f0b98f1 P 451c10fe90714adaa953f0322ee2b2c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451c10fe90714adaa953f0322ee2b2c6, State: Initialized, Role: FOLLOWER
I20260928 22:53:40.072281 15441 raft_consensus.cc:348] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c10fe90714adaa953f0322ee2b2c6" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46159 } } peers { permanent_uuid: "0aaef46e02254a468bdbafbe8a4541a4" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 45867 } } peers { permanent_uuid: "cc8b3a068ab24554a4eaf03174d60bca" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 36181 } }
I20260928 22:53:40.072443 15441 raft_consensus.cc:374] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:40.072500 15441 raft_consensus.cc:729] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8b3a068ab24554a4eaf03174d60bca, State: Initialized, Role: FOLLOWER
I20260928 22:53:40.072535 15440 consensus_queue.cc:260] T 0ace26e572f445ea93fce6a31f0b98f1 P 451c10fe90714adaa953f0322ee2b2c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c10fe90714adaa953f0322ee2b2c6" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46159 } } peers { permanent_uuid: "0aaef46e02254a468bdbafbe8a4541a4" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 45867 } } peers { permanent_uuid: "cc8b3a068ab24554a4eaf03174d60bca" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 36181 } }
I20260928 22:53:40.072629 15441 consensus_queue.cc:260] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c10fe90714adaa953f0322ee2b2c6" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46159 } } peers { permanent_uuid: "0aaef46e02254a468bdbafbe8a4541a4" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 45867 } } peers { permanent_uuid: "cc8b3a068ab24554a4eaf03174d60bca" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 36181 } }
I20260928 22:53:40.072806 15440 ts_tablet_manager.cc:1428] T 0ace26e572f445ea93fce6a31f0b98f1 P 451c10fe90714adaa953f0322ee2b2c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:53:40.072846 15155 heartbeater.cc:503] Master 127.14.14.190:36183 was elected leader, sending a full tablet report...
I20260928 22:53:40.072937 15289 heartbeater.cc:503] Master 127.14.14.190:36183 was elected leader, sending a full tablet report...
I20260928 22:53:40.073195 15441 ts_tablet_manager.cc:1428] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:53:40.073866 15442 tablet_bootstrap.cc:492] T 0ace26e572f445ea93fce6a31f0b98f1 P 0aaef46e02254a468bdbafbe8a4541a4: No bootstrap required, opened a new log
I20260928 22:53:40.073982 15442 ts_tablet_manager.cc:1397] T 0ace26e572f445ea93fce6a31f0b98f1 P 0aaef46e02254a468bdbafbe8a4541a4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 22:53:40.075420 15442 raft_consensus.cc:348] T 0ace26e572f445ea93fce6a31f0b98f1 P 0aaef46e02254a468bdbafbe8a4541a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c10fe90714adaa953f0322ee2b2c6" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46159 } } peers { permanent_uuid: "0aaef46e02254a468bdbafbe8a4541a4" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 45867 } } peers { permanent_uuid: "cc8b3a068ab24554a4eaf03174d60bca" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 36181 } }
I20260928 22:53:40.075614 15442 raft_consensus.cc:374] T 0ace26e572f445ea93fce6a31f0b98f1 P 0aaef46e02254a468bdbafbe8a4541a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:40.075680 15442 raft_consensus.cc:729] T 0ace26e572f445ea93fce6a31f0b98f1 P 0aaef46e02254a468bdbafbe8a4541a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aaef46e02254a468bdbafbe8a4541a4, State: Initialized, Role: FOLLOWER
I20260928 22:53:40.075793 15442 consensus_queue.cc:260] T 0ace26e572f445ea93fce6a31f0b98f1 P 0aaef46e02254a468bdbafbe8a4541a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c10fe90714adaa953f0322ee2b2c6" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46159 } } peers { permanent_uuid: "0aaef46e02254a468bdbafbe8a4541a4" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 45867 } } peers { permanent_uuid: "cc8b3a068ab24554a4eaf03174d60bca" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 36181 } }
I20260928 22:53:40.076081 15442 ts_tablet_manager.cc:1428] T 0ace26e572f445ea93fce6a31f0b98f1 P 0aaef46e02254a468bdbafbe8a4541a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:53:40.076092 15423 heartbeater.cc:503] Master 127.14.14.190:36183 was elected leader, sending a full tablet report...
W20260928 22:53:40.103920 15290 tablet.cc:2369] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:53:40.196329 15156 tablet.cc:2369] T 0ace26e572f445ea93fce6a31f0b98f1 P 451c10fe90714adaa953f0322ee2b2c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:53:40.198128 15447 raft_consensus.cc:482] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:53:40.198235 15447 raft_consensus.cc:504] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c10fe90714adaa953f0322ee2b2c6" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46159 } } peers { permanent_uuid: "0aaef46e02254a468bdbafbe8a4541a4" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 45867 } } peers { permanent_uuid: "cc8b3a068ab24554a4eaf03174d60bca" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 36181 } }
I20260928 22:53:40.198611 15447 leader_election.cc:288] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451c10fe90714adaa953f0322ee2b2c6 (127.14.14.129:46159), 0aaef46e02254a468bdbafbe8a4541a4 (127.14.14.131:45867)
I20260928 22:53:40.201972 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.198864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:46159 (local address 127.14.14.130:40021)
0928 22:53:40.198908 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:40.198921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:40.198995 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:40.199193 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:40.199194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:40.199196 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:40.199271 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:40.199273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.200131 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.200132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:40.200980 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:40.200982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.201721 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.201722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:40.201806 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.201810 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:40.201865 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:53:40.201905 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":19}
I20260928 22:53:40.202029 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.198987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:40021 (local address 127.14.14.129:46159)
0928 22:53:40.199048 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:40.199050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:40.199057 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:53:40.199088 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:40.199089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:40.199096 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:40.199185 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:40.199380 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.200054 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.201072 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.201637 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.201710 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.201718 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:40.201879 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 22:53:40.201917 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:53:40.202142 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.198864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:45867 (local address 127.14.14.130:49665)
0928 22:53:40.199237 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:40.199246 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:40.199295 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:40.199518 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:40.199519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:40.199521 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:40.199597 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:40.199598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.200308 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.200308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:40.201134 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:40.201136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.201872 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.201873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:40.201946 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.201950 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:40.202009 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:53:40.202047 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":184,"thread_start_us":129,"threads_started":1}
I20260928 22:53:40.202143 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.198912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:49665 (local address 127.14.14.131:45867)
0928 22:53:40.198956 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:40.198958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:40.199314 (+ 356us) server_negotiation.cc:407] Connection header received
0928 22:53:40.199346 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:40.199347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:40.199360 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:40.199413 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:40.199667 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.200227 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.201218 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.201768 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.201845 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.201855 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:40.202028 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 22:53:40.202066 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:53:40.202380 15109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ace26e572f445ea93fce6a31f0b98f1" candidate_uuid: "cc8b3a068ab24554a4eaf03174d60bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451c10fe90714adaa953f0322ee2b2c6" is_pre_election: true
I20260928 22:53:40.202580 15109 raft_consensus.cc:2454] T 0ace26e572f445ea93fce6a31f0b98f1 P 451c10fe90714adaa953f0322ee2b2c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8b3a068ab24554a4eaf03174d60bca in term 0.
I20260928 22:53:40.202700 15377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ace26e572f445ea93fce6a31f0b98f1" candidate_uuid: "cc8b3a068ab24554a4eaf03174d60bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aaef46e02254a468bdbafbe8a4541a4" is_pre_election: true
I20260928 22:53:40.202831 15176 leader_election.cc:302] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451c10fe90714adaa953f0322ee2b2c6, cc8b3a068ab24554a4eaf03174d60bca; no voters:
I20260928 22:53:40.202879 15377 raft_consensus.cc:2454] T 0ace26e572f445ea93fce6a31f0b98f1 P 0aaef46e02254a468bdbafbe8a4541a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8b3a068ab24554a4eaf03174d60bca in term 0.
I20260928 22:53:40.203025 15447 raft_consensus.cc:2781] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:53:40.203096 15447 raft_consensus.cc:482] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:53:40.203120 15447 raft_consensus.cc:3037] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:40.203825 15447 raft_consensus.cc:504] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c10fe90714adaa953f0322ee2b2c6" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46159 } } peers { permanent_uuid: "0aaef46e02254a468bdbafbe8a4541a4" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 45867 } } peers { permanent_uuid: "cc8b3a068ab24554a4eaf03174d60bca" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 36181 } }
I20260928 22:53:40.203960 15447 leader_election.cc:288] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [CANDIDATE]: Term 1 election: Requested vote from peers 451c10fe90714adaa953f0322ee2b2c6 (127.14.14.129:46159), 0aaef46e02254a468bdbafbe8a4541a4 (127.14.14.131:45867)
I20260928 22:53:40.204072 15109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ace26e572f445ea93fce6a31f0b98f1" candidate_uuid: "cc8b3a068ab24554a4eaf03174d60bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451c10fe90714adaa953f0322ee2b2c6"
I20260928 22:53:40.204154 15109 raft_consensus.cc:3037] T 0ace26e572f445ea93fce6a31f0b98f1 P 451c10fe90714adaa953f0322ee2b2c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:40.204159 15377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ace26e572f445ea93fce6a31f0b98f1" candidate_uuid: "cc8b3a068ab24554a4eaf03174d60bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aaef46e02254a468bdbafbe8a4541a4"
I20260928 22:53:40.204227 15377 raft_consensus.cc:3037] T 0ace26e572f445ea93fce6a31f0b98f1 P 0aaef46e02254a468bdbafbe8a4541a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:40.204797 15109 raft_consensus.cc:2454] T 0ace26e572f445ea93fce6a31f0b98f1 P 451c10fe90714adaa953f0322ee2b2c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8b3a068ab24554a4eaf03174d60bca in term 1.
I20260928 22:53:40.204955 15377 raft_consensus.cc:2454] T 0ace26e572f445ea93fce6a31f0b98f1 P 0aaef46e02254a468bdbafbe8a4541a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8b3a068ab24554a4eaf03174d60bca in term 1.
I20260928 22:53:40.204996 15176 leader_election.cc:302] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451c10fe90714adaa953f0322ee2b2c6, cc8b3a068ab24554a4eaf03174d60bca; no voters:
I20260928 22:53:40.205096 15447 raft_consensus.cc:2781] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:40.205281 15447 raft_consensus.cc:686] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [term 1 LEADER]: Becoming Leader. State: Replica: cc8b3a068ab24554a4eaf03174d60bca, State: Running, Role: LEADER
I20260928 22:53:40.205391 15447 consensus_queue.cc:237] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c10fe90714adaa953f0322ee2b2c6" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46159 } } peers { permanent_uuid: "0aaef46e02254a468bdbafbe8a4541a4" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 45867 } } peers { permanent_uuid: "cc8b3a068ab24554a4eaf03174d60bca" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 36181 } }
I20260928 22:53:40.206197 14964 catalog_manager.cc:5495] T 0ace26e572f445ea93fce6a31f0b98f1 P cc8b3a068ab24554a4eaf03174d60bca reported cstate change: term changed from 0 to 1, leader changed from <none> to cc8b3a068ab24554a4eaf03174d60bca (127.14.14.130). New cstate: current_term: 1 leader_uuid: "cc8b3a068ab24554a4eaf03174d60bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451c10fe90714adaa953f0322ee2b2c6" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aaef46e02254a468bdbafbe8a4541a4" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 45867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc8b3a068ab24554a4eaf03174d60bca" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 36181 } health_report { overall_health: HEALTHY } } }
Sep 28 22:53:40 dist-test-slave-kzfr krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.129@KRBTEST.COM
I20260928 22:53:40.248593 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.240789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36192 (local address 127.14.14.129:46159)
0928 22:53:40.240861 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:40.240863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:40.240956 (+ 93us) server_negotiation.cc:407] Connection header received
0928 22:53:40.241001 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:40.241001 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:40.241010 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:40.241063 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:40.241690 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.242163 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.242875 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.243051 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.243112 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.245766 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:40.245777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:40.245778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:40.245784 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:40.247485 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:40.247796 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:40.247799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:40.247799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:40.247864 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:40.248127 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:40.248128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:40.248129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:40.248285 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:40.248391 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:40.248460 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 22:53:40.248523 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 22:53:40 dist-test-slave-kzfr krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636020, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 22:53:40 dist-test-slave-kzfr krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636020, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.14.129@KRBTEST.COM
I20260928 22:53:40.271466 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.264126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.14.14.129:46159)
0928 22:53:40.264204 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:40.264207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:40.264287 (+ 80us) server_negotiation.cc:407] Connection header received
0928 22:53:40.264387 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:40.264388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:40.264400 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:40.264474 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:40.265048 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.265549 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.266336 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.266541 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.266627 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.268819 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:40.268820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:40.268821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:40.268829 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:40.270240 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:40.270647 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:40.270649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:40.270649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:40.270713 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:40.270931 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:40.270932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:40.270932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:40.271043 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:40.271173 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:40.271333 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 22:53:40.271397 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 22:53:40 dist-test-slave-kzfr krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260928 22:53:40.282776 15424 tablet.cc:2369] T 0ace26e572f445ea93fce6a31f0b98f1 P 0aaef46e02254a468bdbafbe8a4541a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 22:53:40 dist-test-slave-kzfr krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.129@KRBTEST.COM
I20260928 22:53:40.293812 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:40.286634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36206 (local address 127.14.14.129:46159)
0928 22:53:40.286707 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:40.286710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:40.286791 (+ 81us) server_negotiation.cc:407] Connection header received
0928 22:53:40.286898 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:40.286899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:40.286910 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:40.286978 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:40.287541 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.288039 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.288846 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:40.289042 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:40.289146 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:40.291374 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:40.291376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:40.291376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:40.291384 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:40.292651 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:40.292994 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:40.292995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:40.292996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:40.293056 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:40.293269 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:40.293270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:40.293271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:40.293371 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:40.293476 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:40.293622 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 22:53:40.293696 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 22:53:40.294278 15069 tablet_service.cc:2908] Rejecting scan request for tablet 0ace26e572f445ea93fce6a31f0b98f1: Uninitialized: clean time has not yet been initialized
I20260928 22:53:40.295187 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 15026
I20260928 22:53:40.300280 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 15160
I20260928 22:53:40.305130 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 15294
I20260928 22:53:40.309782 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 14932
2026-09-28T22:53:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2092 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:40 dist-test-slave-kzfr krb5kdc[15464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:40 dist-test-slave-kzfr krb5kdc[15464](info): set up 2 sockets
Sep 28 22:53:40 dist-test-slave-kzfr krb5kdc[15464](info): commencing operation
krb5kdc: starting...
W20260928 22:53:41.341463 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:53:41 dist-test-slave-kzfr krb5kdc[15464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:53:41.464897 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:33011
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:33011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:41.545285 15477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:41.545488 15477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:41.545511 15477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:41.546685 15477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:41.546726 15477 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 22:53:41.546739 15477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:41.546792 15477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:41.546818 15477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:41.546845 15477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:41.548269 15477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:33011
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:33011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:41.548494 15477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:41.548702 15477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:41.553537 15482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:41.553630 15483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:41.553563 15485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:41.553809 15477 server_base.cc:1034] running on GCE node
I20260928 22:53:41.554165 15477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:41.554488 15477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:41.555645 15477 hybrid_clock.cc:648] HybridClock initialized: now 1790636021555632 us; error 38 us; skew 500 ppm
Sep 28 22:53:41 dist-test-slave-kzfr krb5kdc[15464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636021, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:41.562529 15477 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:53:41.563041 15477 webserver.cc:460] Webserver started at http://127.14.14.190:39667/ using document root <none> and password file <none>
I20260928 22:53:41.563277 15477 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:41.563321 15477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:41.563534 15477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:41.564611 15477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "f748720b50ef43ae92123676adb6cc0f"
format_stamp: "Formatted at 2026-09-28 22:53:41 on dist-test-slave-kzfr"
server_key: "701846ccf6bd93988db7176041250df0"
server_key_iv: "5bd1dd80bb7627c053d47fcc281dd98b"
server_key_version: "encryptionkey@0"
I20260928 22:53:41.564988 15477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "f748720b50ef43ae92123676adb6cc0f"
format_stamp: "Formatted at 2026-09-28 22:53:41 on dist-test-slave-kzfr"
server_key: "701846ccf6bd93988db7176041250df0"
server_key_iv: "5bd1dd80bb7627c053d47fcc281dd98b"
server_key_version: "encryptionkey@0"
I20260928 22:53:41.566473 15477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 22:53:41.567226 15492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:41.567426 15477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:41.567538 15477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "f748720b50ef43ae92123676adb6cc0f"
format_stamp: "Formatted at 2026-09-28 22:53:41 on dist-test-slave-kzfr"
server_key: "701846ccf6bd93988db7176041250df0"
server_key_iv: "5bd1dd80bb7627c053d47fcc281dd98b"
server_key_version: "encryptionkey@0"
I20260928 22:53:41.567642 15477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:41.578930 15477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:41.580822 15477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:41.580989 15477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:41.585938 15477 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:33011
I20260928 22:53:41.586030 15554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:33011 every 8 connection(s)
I20260928 22:53:41.586387 15477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:41.587083 15555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:41.589455 15555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f: Bootstrap starting.
I20260928 22:53:41.590073 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 15477
I20260928 22:53:41.590175 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:53:41.590193 15555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:41.590374 14394 external_mini_cluster.cc:1428] Setting key 5a326ce6dc97b9b2a79d3d4a6b0f27da
I20260928 22:53:41.590504 15555 log.cc:824] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:41.591341 15555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f: No bootstrap required, opened a new log
I20260928 22:53:41.593887 15555 raft_consensus.cc:348] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f748720b50ef43ae92123676adb6cc0f" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33011 } }
I20260928 22:53:41.594059 15555 raft_consensus.cc:374] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:41.594094 15555 raft_consensus.cc:729] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f748720b50ef43ae92123676adb6cc0f, State: Initialized, Role: FOLLOWER
I20260928 22:53:41.594235 15555 consensus_queue.cc:260] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f748720b50ef43ae92123676adb6cc0f" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33011 } }
I20260928 22:53:41.594318 15555 raft_consensus.cc:388] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:41.594341 15555 raft_consensus.cc:482] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:41.594412 15555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:41.594980 15555 raft_consensus.cc:504] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f748720b50ef43ae92123676adb6cc0f" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33011 } }
I20260928 22:53:41.595079 15555 leader_election.cc:302] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f748720b50ef43ae92123676adb6cc0f; no voters:
I20260928 22:53:41.595233 15555 leader_election.cc:288] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:41.595299 15560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:41.595468 15555 sys_catalog.cc:564] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:41.595479 15560 raft_consensus.cc:686] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [term 1 LEADER]: Becoming Leader. State: Replica: f748720b50ef43ae92123676adb6cc0f, State: Running, Role: LEADER
I20260928 22:53:41.595583 15560 consensus_queue.cc:237] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f748720b50ef43ae92123676adb6cc0f" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33011 } }
I20260928 22:53:41.596074 15561 sys_catalog.cc:455] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f748720b50ef43ae92123676adb6cc0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f748720b50ef43ae92123676adb6cc0f" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33011 } } }
I20260928 22:53:41.596149 15561 sys_catalog.cc:458] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:41.596279 15562 sys_catalog.cc:455] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f748720b50ef43ae92123676adb6cc0f. Latest consensus state: current_term: 1 leader_uuid: "f748720b50ef43ae92123676adb6cc0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f748720b50ef43ae92123676adb6cc0f" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33011 } } }
I20260928 22:53:41.596347 15562 sys_catalog.cc:458] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:41.596464 15564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:41.597203 15564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:53:41 dist-test-slave-kzfr krb5kdc[15464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:41.599097 15564 catalog_manager.cc:1347] Generated new cluster ID: b71ae4da40044dac82a2646bf39759cb
I20260928 22:53:41.599164 15564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:41.603058 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:41.590961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.14.14.190:33011)
0928 22:53:41.591109 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:41.591119 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:41.591156 (+ 37us) server_negotiation.cc:407] Connection header received
0928 22:53:41.591741 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:41.591750 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:41.591823 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:41.591967 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:41.592570 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.594107 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.595003 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.595816 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.595924 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:41.598804 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:41.598807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:41.598808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:41.598819 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:41.600941 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:41.601359 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:41.601362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:41.601363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:41.601468 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:41.601717 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:41.601718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:41.601718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:41.602022 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:41.602184 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:41.602710 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 22:53:41.602847 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260928 22:53:41.608435 15564 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:41.608953 15564 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:41.618252 15564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f: Generated new TSK 0
I20260928 22:53:41.618425 15564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:53:41.649021 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:33011
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:41.730705 15581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:41.730897 15581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:41.730954 15581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:41.732071 15581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:41.732110 15581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:41.732129 15581 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:53:41.732147 15581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:53:41.733573 15581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.14.190:33011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:41.733775 15581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:41.734087 15581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:41.739285 15586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:41.739275 15589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:41.739270 15587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:41.739307 15581 server_base.cc:1034] running on GCE node
I20260928 22:53:41.739605 15581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:41.739861 15581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:41.741024 15581 hybrid_clock.cc:648] HybridClock initialized: now 1790636021741018 us; error 35 us; skew 500 ppm
Sep 28 22:53:41 dist-test-slave-kzfr krb5kdc[15464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636021, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:41.743677 15581 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:53:41.744097 15581 webserver.cc:460] Webserver started at http://127.14.14.129:41835/ using document root <none> and password file <none>
I20260928 22:53:41.744333 15581 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:41.744400 15581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:41.744557 15581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:41.745674 15581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "7a516ccc3a714f2c9a30e5100373683f"
format_stamp: "Formatted at 2026-09-28 22:53:41 on dist-test-slave-kzfr"
server_key: "afcc8f9a07e788105ac36d5112c56629"
server_key_iv: "3defc04b10c406a6af7b270870dbda1f"
server_key_version: "encryptionkey@0"
I20260928 22:53:41.746037 15581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "7a516ccc3a714f2c9a30e5100373683f"
format_stamp: "Formatted at 2026-09-28 22:53:41 on dist-test-slave-kzfr"
server_key: "afcc8f9a07e788105ac36d5112c56629"
server_key_iv: "3defc04b10c406a6af7b270870dbda1f"
server_key_version: "encryptionkey@0"
I20260928 22:53:41.747359 15581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:53:41.748088 15596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:41.748263 15581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:41.748368 15581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "7a516ccc3a714f2c9a30e5100373683f"
format_stamp: "Formatted at 2026-09-28 22:53:41 on dist-test-slave-kzfr"
server_key: "afcc8f9a07e788105ac36d5112c56629"
server_key_iv: "3defc04b10c406a6af7b270870dbda1f"
server_key_version: "encryptionkey@0"
I20260928 22:53:41.748442 15581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:41.765672 15581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:41.767520 15581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:41.767695 15581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:41.768220 15581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:41.768285 15581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:41.768329 15581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:41.768371 15581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:41.775518 15581 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:39879
I20260928 22:53:41.775560 15711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:39879 every 8 connection(s)
Sep 28 22:53:41 dist-test-slave-kzfr krb5kdc[15464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636021, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:41.775915 15581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:53:41.778668 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:41.768790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:38887 (local address 127.14.14.190:33011)
0928 22:53:41.768830 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:41.768833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:41.769067 (+ 234us) server_negotiation.cc:407] Connection header received
0928 22:53:41.769576 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:41.769577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:41.769612 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:41.769663 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:41.770582 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.771509 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.773053 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.773399 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.773486 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:41.776190 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:41.776192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:41.776192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:41.776199 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:41.777454 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:41.777876 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:41.777877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:41.777877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:41.777926 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:41.778142 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:41.778143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:41.778143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:41.778240 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:41.778293 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:41.778512 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 22:53:41.778577 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:53:41.778800 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:41.768781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33011 (local address 127.14.14.129:38887)
0928 22:53:41.768951 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:41.768985 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:41.769476 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:41.769754 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:41.769757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:41.770178 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:41.770473 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:41.770479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.771618 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.771619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:41.772976 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:41.772979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.773508 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.773510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:41.773766 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:41.774311 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:41.774329 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:41.776074 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:41.777521 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:41.777527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:41.777528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:41.777811 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:41.777985 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:41.777986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:41.777986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:41.778078 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:41.778297 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:41.778301 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:41.778406 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 22:53:41.778486 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 22:53:41.778595 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260928 22:53:41.779317 15712 heartbeater.cc:347] Connected to a master server at 127.14.14.190:33011
I20260928 22:53:41.779397 15712 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:41.779570 15712 heartbeater.cc:511] Master 127.14.14.190:33011 requested a full tablet report, sending...
I20260928 22:53:41.780084 15509 ts_manager.cc:194] Registered new tserver with Master: 7a516ccc3a714f2c9a30e5100373683f (127.14.14.129:39879)
I20260928 22:53:41.780861 15509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:38887
I20260928 22:53:41.783862 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 15581
I20260928 22:53:41.783974 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:53:41.784138 14394 external_mini_cluster.cc:1428] Setting key 85e6a5b02dcda23a70e9477b38ef4c03
I20260928 22:53:41.786098 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:41.780456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:54021 (local address 127.14.14.190:33011)
0928 22:53:41.780491 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:41.780493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:41.780620 (+ 127us) server_negotiation.cc:407] Connection header received
0928 22:53:41.780691 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:41.780692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:41.780702 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:41.780755 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:41.781079 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.781554 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.782278 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.782461 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.782530 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:41.783438 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:41.783439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:41.783440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:41.783446 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:41.784966 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:41.785306 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:41.785308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:41.785308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:41.785360 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:41.785568 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:41.785569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:41.785569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:41.785709 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:41.785769 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:41.785939 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 22:53:41.786009 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:53:41.786132 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:41.780408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33011 (local address 127.14.14.129:54021)
0928 22:53:41.780537 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:41.780547 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:41.780635 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:41.780762 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:41.780762 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:41.780928 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:41.781006 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:41.781007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.781674 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.781675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:41.782182 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:41.782184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.782541 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.782542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:41.782620 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:41.783035 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:41.783040 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:41.783336 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:41.785057 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:41.785058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:41.785058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:41.785222 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:41.785435 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:41.785436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:41.785436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:41.785494 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:41.785772 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:41.785773 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:41.785827 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 22:53:41.785914 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 22:53:41.786011 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:53:41.808952 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:33011
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:41.891870 15721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:41.892113 15721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:41.892153 15721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:41.893343 15721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:41.893422 15721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:41.893467 15721 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:53:41.893512 15721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:53:41.894987 15721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.14.190:33011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:41.895254 15721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:41.895493 15721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:41.900822 15727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:41.900830 15729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:41.900863 15721 server_base.cc:1034] running on GCE node
W20260928 22:53:41.900830 15726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:41.901316 15721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:41.901623 15721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:41.902781 15721 hybrid_clock.cc:648] HybridClock initialized: now 1790636021902771 us; error 28 us; skew 500 ppm
Sep 28 22:53:41 dist-test-slave-kzfr krb5kdc[15464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636021, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:41.905191 15721 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:53:41.905556 15721 webserver.cc:460] Webserver started at http://127.14.14.130:33437/ using document root <none> and password file <none>
I20260928 22:53:41.905890 15721 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:41.905939 15721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:41.906026 15721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:41.907012 15721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "e72fd149e81642c48814bc8e2f30c49a"
format_stamp: "Formatted at 2026-09-28 22:53:41 on dist-test-slave-kzfr"
server_key: "a4852b316880dd6a7154d67a1676dc97"
server_key_iv: "76cf67fa727ca90cbee4adbee17b7f30"
server_key_version: "encryptionkey@0"
I20260928 22:53:41.907303 15721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "e72fd149e81642c48814bc8e2f30c49a"
format_stamp: "Formatted at 2026-09-28 22:53:41 on dist-test-slave-kzfr"
server_key: "a4852b316880dd6a7154d67a1676dc97"
server_key_iv: "76cf67fa727ca90cbee4adbee17b7f30"
server_key_version: "encryptionkey@0"
I20260928 22:53:41.908505 15721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:53:41.909180 15736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:41.909364 15721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:53:41.909431 15721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "e72fd149e81642c48814bc8e2f30c49a"
format_stamp: "Formatted at 2026-09-28 22:53:41 on dist-test-slave-kzfr"
server_key: "a4852b316880dd6a7154d67a1676dc97"
server_key_iv: "76cf67fa727ca90cbee4adbee17b7f30"
server_key_version: "encryptionkey@0"
I20260928 22:53:41.909482 15721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:41.920558 15721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:41.922757 15721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:41.922864 15721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:41.923298 15721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:41.923336 15721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:41.923359 15721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:41.923398 15721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:41.930294 15721 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:34069
I20260928 22:53:41.930368 15851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:34069 every 8 connection(s)
Sep 28 22:53:41 dist-test-slave-kzfr krb5kdc[15464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636021, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:41.930696 15721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:53:41.933275 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:41.923841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:47559 (local address 127.14.14.190:33011)
0928 22:53:41.923922 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:41.923925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:41.924069 (+ 144us) server_negotiation.cc:407] Connection header received
0928 22:53:41.924549 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:41.924551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:41.924564 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:41.924644 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:41.925521 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.926393 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.927997 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.928253 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.928340 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:41.931113 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:41.931114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:41.931114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:41.931125 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:41.932214 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:41.932555 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:41.932555 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:41.932556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:41.932604 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:41.932809 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:41.932810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:41.932810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:41.932903 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:41.932953 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:41.933160 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 22:53:41.933215 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 22:53:41.933441 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:41.923736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33011 (local address 127.14.14.130:47559)
0928 22:53:41.923939 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:41.923977 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:41.924450 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:41.924756 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:41.924760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:41.925112 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:41.925405 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:41.925412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.926511 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.926512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:41.927916 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:41.927920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.928352 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.928353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:41.928497 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:41.929094 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:41.929105 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:41.931014 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:41.932279 (+ 1265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:41.932284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:41.932285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:41.932489 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:41.932664 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:41.932664 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:41.932664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:41.932756 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:41.932956 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:41.932960 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:41.933059 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 22:53:41.933142 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:53:41.933253 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":107,"threads_started":1}
I20260928 22:53:41.933945 15852 heartbeater.cc:347] Connected to a master server at 127.14.14.190:33011
I20260928 22:53:41.934041 15852 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:41.934238 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 15721
I20260928 22:53:41.934334 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:53:41.934512 14394 external_mini_cluster.cc:1428] Setting key 8eaf011b42aaf7405b7efc503c5cf6bd
I20260928 22:53:41.934845 15852 heartbeater.cc:511] Master 127.14.14.190:33011 requested a full tablet report, sending...
I20260928 22:53:41.935225 15509 ts_manager.cc:194] Registered new tserver with Master: e72fd149e81642c48814bc8e2f30c49a (127.14.14.130:34069)
I20260928 22:53:41.935695 15509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:47559
I20260928 22:53:41.941509 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:41.934924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33011 (local address 127.14.14.130:37969)
0928 22:53:41.935118 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:41.935129 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:41.935178 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:41.935599 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:41.935600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:41.935778 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:41.935849 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:41.935851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.936761 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.936763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:41.937264 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:41.937266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.937692 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.937693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:41.937773 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:41.938121 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:41.938126 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:41.938545 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:41.940509 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:41.940510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:41.940510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:41.940672 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:41.940890 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:41.940890 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:41.940891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:41.940940 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:41.941274 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:41.941274 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:41.941318 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 22:53:41.941367 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 22:53:41.941439 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":78,"threads_started":1}
I20260928 22:53:41.941663 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:41.935093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:37969 (local address 127.14.14.190:33011)
0928 22:53:41.935129 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:41.935131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:41.935166 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:53:41.935220 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:41.935220 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:41.935230 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:41.935285 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:41.935947 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.936664 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.937359 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:41.937561 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:41.937689 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:41.938635 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:41.938636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:41.938636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:41.938642 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:41.940410 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:41.940747 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:41.940749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:41.940750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:41.940819 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:41.940999 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:41.941000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:41.941000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:41.941205 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:41.941280 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:41.941422 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 22:53:41.941506 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:53:41.961215 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:33011
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:42.042970 15861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:42.043222 15861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:42.043258 15861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:42.044452 15861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:42.044494 15861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:42.044514 15861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:53:42.044576 15861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:53:42.046048 15861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.14.190:33011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:42.046293 15861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:42.046519 15861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:42.051656 15867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:42.051699 15869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:42.051704 15866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:42.052101 15861 server_base.cc:1034] running on GCE node
I20260928 22:53:42.052349 15861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:42.052560 15861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:42.053653 15861 hybrid_clock.cc:648] HybridClock initialized: now 1790636022053641 us; error 27 us; skew 500 ppm
Sep 28 22:53:42 dist-test-slave-kzfr krb5kdc[15464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636022, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:42.056123 15861 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:53:42.056546 15861 webserver.cc:460] Webserver started at http://127.14.14.131:34709/ using document root <none> and password file <none>
I20260928 22:53:42.056784 15861 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:42.056854 15861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:42.057009 15861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:42.058153 15861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "28af8ebe1222484db27e886c54a7ea82"
format_stamp: "Formatted at 2026-09-28 22:53:42 on dist-test-slave-kzfr"
server_key: "95dd56a3c32d784fc89dcb00a6fc86ec"
server_key_iv: "97b700133afef3e11d4471cc32907f3e"
server_key_version: "encryptionkey@0"
I20260928 22:53:42.058516 15861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "28af8ebe1222484db27e886c54a7ea82"
format_stamp: "Formatted at 2026-09-28 22:53:42 on dist-test-slave-kzfr"
server_key: "95dd56a3c32d784fc89dcb00a6fc86ec"
server_key_iv: "97b700133afef3e11d4471cc32907f3e"
server_key_version: "encryptionkey@0"
I20260928 22:53:42.059823 15861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:53:42.060526 15876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:42.060727 15861 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:42.060832 15861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "28af8ebe1222484db27e886c54a7ea82"
format_stamp: "Formatted at 2026-09-28 22:53:42 on dist-test-slave-kzfr"
server_key: "95dd56a3c32d784fc89dcb00a6fc86ec"
server_key_iv: "97b700133afef3e11d4471cc32907f3e"
server_key_version: "encryptionkey@0"
I20260928 22:53:42.060906 15861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:42.089967 15861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:42.092412 15861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:42.092568 15861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:42.093151 15861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:42.093222 15861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:42.093261 15861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:42.093291 15861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:42.100265 15861 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:41387
I20260928 22:53:42.100342 15991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:41387 every 8 connection(s)
Sep 28 22:53:42 dist-test-slave-kzfr krb5kdc[15464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636022, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:42.100718 15861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:53:42.103854 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.093706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:41949 (local address 127.14.14.190:33011)
0928 22:53:42.093770 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.093772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.094019 (+ 247us) server_negotiation.cc:407] Connection header received
0928 22:53:42.094532 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.094534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.094549 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.094664 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:42.095544 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.096429 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.098146 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.098378 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.098474 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.101191 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:42.101192 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:42.101192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:42.101200 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:42.102730 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:42.103129 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:42.103129 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:42.103130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:42.103178 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:42.103384 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:42.103384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:42.103384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:42.103481 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:42.103530 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.103736 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.103793 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:53:42.104097 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.093720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33011 (local address 127.14.14.131:41949)
0928 22:53:42.093905 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:42.093946 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:42.094427 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:42.094741 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:42.094746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:42.095136 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:42.095441 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.095448 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.096549 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.096550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.098048 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.098051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.098484 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.098485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.098633 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.099174 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:42.099183 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:42.101089 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:42.102819 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:42.102826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:42.102828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:42.103062 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:42.103235 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:42.103236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:42.103236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:42.103331 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:42.103533 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:42.103537 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:42.103647 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 22:53:42.103727 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 22:53:42.103839 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":110,"threads_started":1}
I20260928 22:53:42.104596 15992 heartbeater.cc:347] Connected to a master server at 127.14.14.190:33011
I20260928 22:53:42.104700 15992 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:42.104887 15992 heartbeater.cc:511] Master 127.14.14.190:33011 requested a full tablet report, sending...
I20260928 22:53:42.105283 15508 ts_manager.cc:194] Registered new tserver with Master: 28af8ebe1222484db27e886c54a7ea82 (127.14.14.131:41387)
I20260928 22:53:42.105795 15508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:41949
I20260928 22:53:42.106145 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 15861
I20260928 22:53:42.106231 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:53:42.106415 14394 external_mini_cluster.cc:1428] Setting key bff77c89e9075265e2b7e12a8cd6acc6
I20260928 22:53:42.107129 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:53:42.111361 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.105660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:49177 (local address 127.14.14.190:33011)
0928 22:53:42.105704 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.105706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.105840 (+ 134us) server_negotiation.cc:407] Connection header received
0928 22:53:42.105865 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.105866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.105876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.105936 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:42.106369 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.106917 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.107672 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.107876 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.107946 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.108843 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:42.108844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:42.108844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:42.108851 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:42.110208 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:42.110562 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:42.110563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:42.110563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:42.110638 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:42.110840 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:42.110841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:42.110841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:42.110949 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:42.111024 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.111204 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.111273 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:53:42.111374 15998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.105564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33011 (local address 127.14.14.131:49177)
0928 22:53:42.105758 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:42.105776 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:42.105831 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:42.105998 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:42.105999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:42.106226 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:42.106300 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.106302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.107017 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.107018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.107586 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.107587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.107956 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.107957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.108040 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.108455 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:42.108459 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:42.108740 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:42.110306 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:42.110307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:42.110307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:42.110476 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:42.110705 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:42.110706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:42.110706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:42.110765 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:42.111016 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:42.111017 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:42.111072 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 22:53:42.111174 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 22:53:42.111274 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":115,"threads_started":1}
Sep 28 22:53:42 dist-test-slave-kzfr krb5kdc[15464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636022, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 22:53:42 dist-test-slave-kzfr krb5kdc[15464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636022, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:42.129423 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.122986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.14.14.190:33011)
0928 22:53:42.123044 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.123046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.123072 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:53:42.123129 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.123129 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.123137 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.123185 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:42.123711 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.124172 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.124859 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.125027 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.125073 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.127130 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:42.127131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:42.127131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:42.127137 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:42.128445 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:42.128742 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:42.128743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:42.128743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:42.128789 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:42.128949 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:42.128950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:42.128950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:42.129037 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:42.129105 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.129218 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.129284 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:53:42.130971 15508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:53:42.132503 15508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:53:42.139148 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.134836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:41387 (local address 127.0.0.1:45312)
0928 22:53:42.135016 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:42.135059 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:42.135122 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:42.135607 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:42.135608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:42.135610 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:42.135664 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.135666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.136994 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.136995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.137941 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.137943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.138853 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.138854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.138920 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.138939 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 22:53:42.138976 (+ 37us) client_negotiation.cc:240] Negotiation successful
0928 22:53:42.139021 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260928 22:53:42.139309 16009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.134837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:39879 (local address 127.0.0.1:39182)
0928 22:53:42.135221 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:42.135230 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:42.135277 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:42.135519 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:42.135522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:42.135524 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:42.135586 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.135589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.136984 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.136985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.138158 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.138160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.138822 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.138823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.138919 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.138931 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 22:53:42.139022 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 22:53:42.139062 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":31,"thread_start_us":55,"threads_started":1}
I20260928 22:53:42.139321 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.135125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45312 (local address 127.14.14.131:41387)
0928 22:53:42.135272 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.135277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.135285 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:53:42.135323 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.135325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.135341 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.135610 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:42.135752 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.136886 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.138018 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.138748 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.139108 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.139166 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.139188 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.139226 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260928 22:53:42.139328 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.135314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52640 (local address 127.14.14.130:34069)
0928 22:53:42.135495 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.135501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.135549 (+ 48us) server_negotiation.cc:407] Connection header received
0928 22:53:42.135790 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.135793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.135811 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.135885 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:42.135971 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.136649 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.138136 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.138815 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.138873 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.138927 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.139187 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.139226 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260928 22:53:42.139437 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.134848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:34069 (local address 127.0.0.1:52640)
0928 22:53:42.135496 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:42.135507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:42.135703 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:42.135881 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:42.135882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:42.135883 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:42.135918 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.135919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.136739 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.136740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.138034 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.138036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.138957 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.138958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.139010 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.139013 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:53:42.139059 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 22:53:42.139099 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":30,"thread_start_us":98,"threads_started":1}
I20260928 22:53:42.139179 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.135044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.14.14.129:39879)
0928 22:53:42.135211 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.135216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.135271 (+ 55us) server_negotiation.cc:407] Connection header received
0928 22:53:42.135339 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.135341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.135354 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.135431 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:42.135818 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.136909 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.138240 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.138725 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.138813 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.138873 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.139036 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.139077 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260928 22:53:42.140153 15786 tablet_service.cc:1467] Processing CreateTablet for tablet 2ec2d9ea515344288412e3e3c8bc7351 (DEFAULT_TABLE table=test-table [id=4ba18e867ac84274b233f5e0b8310089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:42.140156 15926 tablet_service.cc:1467] Processing CreateTablet for tablet 2ec2d9ea515344288412e3e3c8bc7351 (DEFAULT_TABLE table=test-table [id=4ba18e867ac84274b233f5e0b8310089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:42.140352 15646 tablet_service.cc:1467] Processing CreateTablet for tablet 2ec2d9ea515344288412e3e3c8bc7351 (DEFAULT_TABLE table=test-table [id=4ba18e867ac84274b233f5e0b8310089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:42.140523 15926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec2d9ea515344288412e3e3c8bc7351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:42.140523 15786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec2d9ea515344288412e3e3c8bc7351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:42.140722 15646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec2d9ea515344288412e3e3c8bc7351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:42.143702 16015 tablet_bootstrap.cc:492] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f: Bootstrap starting.
I20260928 22:53:42.143996 16016 tablet_bootstrap.cc:492] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82: Bootstrap starting.
I20260928 22:53:42.144248 16017 tablet_bootstrap.cc:492] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a: Bootstrap starting.
I20260928 22:53:42.144376 16015 tablet_bootstrap.cc:654] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:42.144771 16015 log.cc:824] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:42.144892 16017 tablet_bootstrap.cc:654] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:42.145043 16016 tablet_bootstrap.cc:654] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:42.145242 16017 log.cc:824] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:42.145416 16016 log.cc:824] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:42.145686 16015 tablet_bootstrap.cc:492] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f: No bootstrap required, opened a new log
I20260928 22:53:42.145783 16015 ts_tablet_manager.cc:1397] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:53:42.146036 16017 tablet_bootstrap.cc:492] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a: No bootstrap required, opened a new log
I20260928 22:53:42.146124 16017 ts_tablet_manager.cc:1397] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:53:42.146171 16016 tablet_bootstrap.cc:492] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82: No bootstrap required, opened a new log
I20260928 22:53:42.146240 16016 ts_tablet_manager.cc:1397] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:53:42.147464 16017 raft_consensus.cc:348] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } }
I20260928 22:53:42.147616 16017 raft_consensus.cc:374] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:42.147660 16017 raft_consensus.cc:729] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72fd149e81642c48814bc8e2f30c49a, State: Initialized, Role: FOLLOWER
I20260928 22:53:42.147766 16017 consensus_queue.cc:260] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } }
I20260928 22:53:42.147836 16015 raft_consensus.cc:348] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } }
I20260928 22:53:42.147994 16015 raft_consensus.cc:374] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:42.148041 16015 raft_consensus.cc:729] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a516ccc3a714f2c9a30e5100373683f, State: Initialized, Role: FOLLOWER
I20260928 22:53:42.148083 16017 ts_tablet_manager.cc:1428] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:53:42.148232 15852 heartbeater.cc:503] Master 127.14.14.190:33011 was elected leader, sending a full tablet report...
I20260928 22:53:42.148190 16016 raft_consensus.cc:348] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } }
I20260928 22:53:42.148331 16016 raft_consensus.cc:374] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:42.148372 16016 raft_consensus.cc:729] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28af8ebe1222484db27e886c54a7ea82, State: Initialized, Role: FOLLOWER
I20260928 22:53:42.148478 16016 consensus_queue.cc:260] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } }
I20260928 22:53:42.148173 16015 consensus_queue.cc:260] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } }
I20260928 22:53:42.148749 16016 ts_tablet_manager.cc:1428] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 22:53:42.148787 15992 heartbeater.cc:503] Master 127.14.14.190:33011 was elected leader, sending a full tablet report...
I20260928 22:53:42.148797 16015 ts_tablet_manager.cc:1428] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:53:42.148902 15712 heartbeater.cc:503] Master 127.14.14.190:33011 was elected leader, sending a full tablet report...
W20260928 22:53:42.181248 15853 tablet.cc:2369] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:53:42.196101 16021 raft_consensus.cc:482] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:53:42.196280 16021 raft_consensus.cc:504] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } }
I20260928 22:53:42.196709 16021 leader_election.cc:288] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28af8ebe1222484db27e886c54a7ea82 (127.14.14.131:41387), 7a516ccc3a714f2c9a30e5100373683f (127.14.14.129:39879)
I20260928 22:53:42.200162 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.196900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:39879 (local address 127.14.14.130:51449)
0928 22:53:42.196964 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:42.196976 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:42.197033 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:42.197206 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:42.197206 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:42.197208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:42.197279 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.197281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.198094 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.198095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.199373 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.199375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.199913 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.199914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.199974 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.199977 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:53:42.200047 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 22:53:42.200089 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":52}
I20260928 22:53:42.200173 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.197022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:51449 (local address 127.14.14.129:39879)
0928 22:53:42.197072 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.197074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.197082 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:53:42.197108 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.197109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.197136 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.197203 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:42.197397 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.198007 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.199461 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.199839 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.199900 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.199907 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.200057 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.200099 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:53:42.200517 15666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec2d9ea515344288412e3e3c8bc7351" candidate_uuid: "e72fd149e81642c48814bc8e2f30c49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a516ccc3a714f2c9a30e5100373683f" is_pre_election: true
I20260928 22:53:42.200696 15666 raft_consensus.cc:2454] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72fd149e81642c48814bc8e2f30c49a in term 0.
I20260928 22:53:42.200730 16024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.196902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:41387 (local address 127.14.14.130:55667)
0928 22:53:42.197194 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:42.197210 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:42.197287 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:42.197421 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:42.197422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:42.197424 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:42.197489 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.197490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.198143 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.198144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.199373 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.199375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.200462 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.200463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.200538 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.200543 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:53:42.200602 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:53:42.200643 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":125,"thread_start_us":113,"threads_started":1}
I20260928 22:53:42.200778 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.197003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:55667 (local address 127.14.14.131:41387)
0928 22:53:42.197061 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.197063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.197275 (+ 212us) server_negotiation.cc:407] Connection header received
0928 22:53:42.197342 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.197344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.197368 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.197417 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:42.197551 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.198075 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.199938 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.200376 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.200453 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.200459 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.200621 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.200694 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:53:42.200948 15738 leader_election.cc:302] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a516ccc3a714f2c9a30e5100373683f, e72fd149e81642c48814bc8e2f30c49a; no voters:
I20260928 22:53:42.201068 16021 raft_consensus.cc:2781] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:53:42.201146 16021 raft_consensus.cc:482] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:53:42.201181 16021 raft_consensus.cc:3037] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:42.201160 15946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec2d9ea515344288412e3e3c8bc7351" candidate_uuid: "e72fd149e81642c48814bc8e2f30c49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28af8ebe1222484db27e886c54a7ea82" is_pre_election: true
I20260928 22:53:42.201319 15946 raft_consensus.cc:2454] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72fd149e81642c48814bc8e2f30c49a in term 0.
I20260928 22:53:42.202328 16021 raft_consensus.cc:504] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } }
I20260928 22:53:42.202485 16021 leader_election.cc:288] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [CANDIDATE]: Term 1 election: Requested vote from peers 28af8ebe1222484db27e886c54a7ea82 (127.14.14.131:41387), 7a516ccc3a714f2c9a30e5100373683f (127.14.14.129:39879)
I20260928 22:53:42.202639 15946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec2d9ea515344288412e3e3c8bc7351" candidate_uuid: "e72fd149e81642c48814bc8e2f30c49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28af8ebe1222484db27e886c54a7ea82"
I20260928 22:53:42.202625 15666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec2d9ea515344288412e3e3c8bc7351" candidate_uuid: "e72fd149e81642c48814bc8e2f30c49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a516ccc3a714f2c9a30e5100373683f"
I20260928 22:53:42.202706 15946 raft_consensus.cc:3037] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:42.202706 15666 raft_consensus.cc:3037] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:42.203586 15666 raft_consensus.cc:2454] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72fd149e81642c48814bc8e2f30c49a in term 1.
I20260928 22:53:42.203586 15946 raft_consensus.cc:2454] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72fd149e81642c48814bc8e2f30c49a in term 1.
I20260928 22:53:42.203752 15738 leader_election.cc:302] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a516ccc3a714f2c9a30e5100373683f, e72fd149e81642c48814bc8e2f30c49a; no voters:
I20260928 22:53:42.203850 16021 raft_consensus.cc:2781] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:42.204003 16021 raft_consensus.cc:686] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [term 1 LEADER]: Becoming Leader. State: Replica: e72fd149e81642c48814bc8e2f30c49a, State: Running, Role: LEADER
I20260928 22:53:42.204098 16021 consensus_queue.cc:237] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } }
I20260928 22:53:42.204887 15509 catalog_manager.cc:5495] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a reported cstate change: term changed from 0 to 1, leader changed from <none> to e72fd149e81642c48814bc8e2f30c49a (127.14.14.130). New cstate: current_term: 1 leader_uuid: "e72fd149e81642c48814bc8e2f30c49a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } health_report { overall_health: HEALTHY } } }
I20260928 22:53:42.214080 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.211777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.14.14.130:34069)
0928 22:53:42.211846 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.211847 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.211855 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:53:42.211881 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.211882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.211896 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.211984 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:42.212137 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.212577 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.213375 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.213553 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.213664 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.213825 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:42.213911 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:42.213968 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.214013 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.214030 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 22:53:42.215746 15666 raft_consensus.cc:1261] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f [term 1 FOLLOWER]: Refusing update from remote peer e72fd149e81642c48814bc8e2f30c49a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:42.215752 15946 raft_consensus.cc:1261] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82 [term 1 FOLLOWER]: Refusing update from remote peer e72fd149e81642c48814bc8e2f30c49a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:42.216073 16021 consensus_queue.cc:1035] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [LEADER]: Connected to new peer: Peer: permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:42.216244 16025 consensus_queue.cc:1035] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:42.218577 16028 mvcc.cc:205] Tried to move back new op lower bound from 7334445146993160192 to 7334445146948558848. Current Snapshot: MvccSnapshot[applied={T|T < 7334445146993160192}]
I20260928 22:53:42.218775 16030 mvcc.cc:205] Tried to move back new op lower bound from 7334445146993160192 to 7334445146948558848. Current Snapshot: MvccSnapshot[applied={T|T < 7334445146993160192}]
I20260928 22:53:42.221853 15508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:42.221961 15508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:42.222690 15508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f: Sending DeleteTablet for 3 replicas of tablet 2ec2d9ea515344288412e3e3c8bc7351
I20260928 22:53:42.223033 15786 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ec2d9ea515344288412e3e3c8bc7351 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:42 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:52640
I20260928 22:53:42.223085 15926 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ec2d9ea515344288412e3e3c8bc7351 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:42 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:45312
I20260928 22:53:42.223084 15646 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ec2d9ea515344288412e3e3c8bc7351 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:42 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:39182
I20260928 22:53:42.223335 16039 tablet_replica.cc:330] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82: stopping tablet replica
I20260928 22:53:42.223349 16038 tablet_replica.cc:330] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a: stopping tablet replica
I20260928 22:53:42.223482 16038 raft_consensus.cc:2229] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:53:42.223497 16039 raft_consensus.cc:2229] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:42.223523 15508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 22:53:42.223609 16039 raft_consensus.cc:2258] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 22:53:42.223671 15508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:53:42.223848 16038 raft_consensus.cc:2258] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:42.224490 16038 ts_tablet_manager.cc:1905] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:42.224822 15646 tablet_service.cc:1467] Processing CreateTablet for tablet 81c8ca158952470e862e5c30b6d5e920 (DEFAULT_TABLE table=test-table [id=c75740fb766a499e9e843705fb6309d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:42.224859 15926 tablet_service.cc:1467] Processing CreateTablet for tablet 81c8ca158952470e862e5c30b6d5e920 (DEFAULT_TABLE table=test-table [id=c75740fb766a499e9e843705fb6309d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:42.224949 15646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c8ca158952470e862e5c30b6d5e920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:42.224962 15926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c8ca158952470e862e5c30b6d5e920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:42.225796 15786 tablet_service.cc:1467] Processing CreateTablet for tablet 81c8ca158952470e862e5c30b6d5e920 (DEFAULT_TABLE table=test-table [id=c75740fb766a499e9e843705fb6309d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:42.225899 15786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c8ca158952470e862e5c30b6d5e920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:42.226377 16015 tablet_bootstrap.cc:492] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f: Bootstrap starting.
I20260928 22:53:42.227077 16015 tablet_bootstrap.cc:654] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:42.227257 16017 tablet_bootstrap.cc:492] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a: Bootstrap starting.
I20260928 22:53:42.227591 16016 tablet_bootstrap.cc:492] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82: Bootstrap starting.
I20260928 22:53:42.227665 16017 tablet_bootstrap.cc:654] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:42.228003 16016 tablet_bootstrap.cc:654] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:42.228022 16039 ts_tablet_manager.cc:1905] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:42.228360 16038 ts_tablet_manager.cc:1918] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:53:42.228469 16038 log.cc:1196] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/2ec2d9ea515344288412e3e3c8bc7351
I20260928 22:53:42.228785 16038 ts_tablet_manager.cc:1939] T 2ec2d9ea515344288412e3e3c8bc7351 P e72fd149e81642c48814bc8e2f30c49a: Deleting consensus metadata
I20260928 22:53:42.229220 15495 catalog_manager.cc:4841] TS e72fd149e81642c48814bc8e2f30c49a (127.14.14.130:34069): tablet 2ec2d9ea515344288412e3e3c8bc7351 (table test-table [id=4ba18e867ac84274b233f5e0b8310089]) successfully deleted
I20260928 22:53:42.229660 16039 ts_tablet_manager.cc:1918] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:53:42.229728 16039 log.cc:1196] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/2ec2d9ea515344288412e3e3c8bc7351
I20260928 22:53:42.229745 16017 tablet_bootstrap.cc:492] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a: No bootstrap required, opened a new log
I20260928 22:53:42.229789 16017 ts_tablet_manager.cc:1397] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 22:53:42.229862 16040 tablet_replica.cc:330] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f: stopping tablet replica
I20260928 22:53:42.229965 16040 raft_consensus.cc:2229] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:42.229939 16017 raft_consensus.cc:348] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.229985 16039 ts_tablet_manager.cc:1939] T 2ec2d9ea515344288412e3e3c8bc7351 P 28af8ebe1222484db27e886c54a7ea82: Deleting consensus metadata
I20260928 22:53:42.230005 16017 raft_consensus.cc:374] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:42.230027 16017 raft_consensus.cc:729] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72fd149e81642c48814bc8e2f30c49a, State: Initialized, Role: FOLLOWER
I20260928 22:53:42.230072 16040 raft_consensus.cc:2258] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:42.230067 16017 consensus_queue.cc:260] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.230166 16017 ts_tablet_manager.cc:1428] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:42.230545 15496 catalog_manager.cc:4841] TS 28af8ebe1222484db27e886c54a7ea82 (127.14.14.131:41387): tablet 2ec2d9ea515344288412e3e3c8bc7351 (table test-table [id=4ba18e867ac84274b233f5e0b8310089]) successfully deleted
I20260928 22:53:42.230855 16016 tablet_bootstrap.cc:492] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82: No bootstrap required, opened a new log
I20260928 22:53:42.230904 16016 ts_tablet_manager.cc:1397] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 22:53:42.231021 16040 ts_tablet_manager.cc:1905] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:42.231037 16016 raft_consensus.cc:348] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.231098 16016 raft_consensus.cc:374] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:42.231117 16016 raft_consensus.cc:729] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28af8ebe1222484db27e886c54a7ea82, State: Initialized, Role: FOLLOWER
I20260928 22:53:42.231158 16016 consensus_queue.cc:260] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.231247 16016 ts_tablet_manager.cc:1428] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:42.232946 16040 ts_tablet_manager.cc:1918] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:53:42.233031 16040 log.cc:1196] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/wal/wals/2ec2d9ea515344288412e3e3c8bc7351
I20260928 22:53:42.233264 16040 ts_tablet_manager.cc:1939] T 2ec2d9ea515344288412e3e3c8bc7351 P 7a516ccc3a714f2c9a30e5100373683f: Deleting consensus metadata
I20260928 22:53:42.233681 15494 catalog_manager.cc:4841] TS 7a516ccc3a714f2c9a30e5100373683f (127.14.14.129:39879): tablet 2ec2d9ea515344288412e3e3c8bc7351 (table test-table [id=4ba18e867ac84274b233f5e0b8310089]) successfully deleted
I20260928 22:53:42.237972 16015 tablet_bootstrap.cc:492] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f: No bootstrap required, opened a new log
I20260928 22:53:42.238027 16015 ts_tablet_manager.cc:1397] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260928 22:53:42.238196 16015 raft_consensus.cc:348] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.238268 16015 raft_consensus.cc:374] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:42.238299 16015 raft_consensus.cc:729] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a516ccc3a714f2c9a30e5100373683f, State: Initialized, Role: FOLLOWER
I20260928 22:53:42.238336 16015 consensus_queue.cc:260] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.238420 16015 ts_tablet_manager.cc:1428] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:42.272847 16023 raft_consensus.cc:482] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:53:42.272979 16023 raft_consensus.cc:504] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.273367 16023 leader_election.cc:288] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72fd149e81642c48814bc8e2f30c49a (127.14.14.130:34069), 28af8ebe1222484db27e886c54a7ea82 (127.14.14.131:41387)
I20260928 22:53:42.276631 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.273726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:37727 (local address 127.14.14.130:34069)
0928 22:53:42.273774 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.273776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.273989 (+ 213us) server_negotiation.cc:407] Connection header received
0928 22:53:42.274017 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.274019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.274029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.274084 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:42.274279 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.274950 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.275902 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.276264 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.276351 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.276360 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.276511 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.276555 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:53:42.276633 15713 tablet.cc:2369] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:53:42.276942 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.273535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:34069 (local address 127.14.14.129:37727)
0928 22:53:42.273750 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:42.273763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:42.273836 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:42.274111 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:42.274112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:42.274115 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:42.274197 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.274200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.275019 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.275020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.275809 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.275811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.276350 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.276351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.276427 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.276430 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:53:42.276504 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 22:53:42.276872 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":13,"thread_start_us":101,"threads_started":1}
I20260928 22:53:42.277263 15806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c8ca158952470e862e5c30b6d5e920" candidate_uuid: "7a516ccc3a714f2c9a30e5100373683f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72fd149e81642c48814bc8e2f30c49a" is_pre_election: true
I20260928 22:53:42.277380 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.273536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:41387 (local address 127.14.14.129:43129)
0928 22:53:42.273611 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:42.273623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:42.273665 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:42.273956 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:42.273957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:42.273959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:42.274035 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.274037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.274981 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.274982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.276272 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.276275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.277126 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.277127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.277213 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.277217 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:42.277273 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:53:42.277314 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:53:42.277408 15806 raft_consensus.cc:2454] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a516ccc3a714f2c9a30e5100373683f in term 0.
I20260928 22:53:42.277571 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.273790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:43129 (local address 127.14.14.131:41387)
0928 22:53:42.273837 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.273839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.273846 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:53:42.273879 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.273879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.273886 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.273951 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:42.274116 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.274888 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.276390 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.277031 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.277117 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.277125 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.277459 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.277510 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:53:42.277604 15599 leader_election.cc:302] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a516ccc3a714f2c9a30e5100373683f, e72fd149e81642c48814bc8e2f30c49a; no voters:
I20260928 22:53:42.277726 16023 raft_consensus.cc:2781] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:53:42.277788 16023 raft_consensus.cc:482] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:53:42.277814 16023 raft_consensus.cc:3037] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:42.278003 15946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c8ca158952470e862e5c30b6d5e920" candidate_uuid: "7a516ccc3a714f2c9a30e5100373683f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28af8ebe1222484db27e886c54a7ea82" is_pre_election: true
I20260928 22:53:42.278088 15946 raft_consensus.cc:2454] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a516ccc3a714f2c9a30e5100373683f in term 0.
I20260928 22:53:42.278631 16023 raft_consensus.cc:504] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.278774 16023 leader_election.cc:288] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [CANDIDATE]: Term 1 election: Requested vote from peers e72fd149e81642c48814bc8e2f30c49a (127.14.14.130:34069), 28af8ebe1222484db27e886c54a7ea82 (127.14.14.131:41387)
I20260928 22:53:42.278921 15806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c8ca158952470e862e5c30b6d5e920" candidate_uuid: "7a516ccc3a714f2c9a30e5100373683f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72fd149e81642c48814bc8e2f30c49a"
I20260928 22:53:42.278959 15946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c8ca158952470e862e5c30b6d5e920" candidate_uuid: "7a516ccc3a714f2c9a30e5100373683f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28af8ebe1222484db27e886c54a7ea82"
I20260928 22:53:42.279001 15806 raft_consensus.cc:3037] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:42.279017 15946 raft_consensus.cc:3037] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:42.279799 15946 raft_consensus.cc:2454] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a516ccc3a714f2c9a30e5100373683f in term 1.
I20260928 22:53:42.279824 15806 raft_consensus.cc:2454] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a516ccc3a714f2c9a30e5100373683f in term 1.
I20260928 22:53:42.279980 15600 leader_election.cc:302] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28af8ebe1222484db27e886c54a7ea82, 7a516ccc3a714f2c9a30e5100373683f; no voters:
I20260928 22:53:42.280061 16023 raft_consensus.cc:2781] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:42.280169 16023 raft_consensus.cc:686] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [term 1 LEADER]: Becoming Leader. State: Replica: 7a516ccc3a714f2c9a30e5100373683f, State: Running, Role: LEADER
I20260928 22:53:42.280236 16023 consensus_queue.cc:237] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.280962 15508 catalog_manager.cc:5495] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a516ccc3a714f2c9a30e5100373683f (127.14.14.129). New cstate: current_term: 1 leader_uuid: "7a516ccc3a714f2c9a30e5100373683f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } health_report { overall_health: UNKNOWN } } }
Sep 28 22:53:42 dist-test-slave-kzfr krb5kdc[15464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636021, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:42.309882 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.302400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.14.14.190:33011)
0928 22:53:42.302473 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.302476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.302639 (+ 163us) server_negotiation.cc:407] Connection header received
0928 22:53:42.302708 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.302709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.302719 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.302794 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:42.303236 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.303725 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.304486 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.304721 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.304805 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.307258 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:42.307260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:42.307260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:42.307266 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:42.308643 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:42.309136 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:42.309137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:42.309137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:42.309184 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:42.309394 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:42.309395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:42.309395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:42.309492 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:42.309540 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.309749 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.309818 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 22:53:42.309991 16052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.302364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33011 (local address 127.0.0.1:38654)
0928 22:53:42.302553 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:42.302562 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:42.302646 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:42.302799 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:42.302800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:42.303095 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:42.303151 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.303153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.303807 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.303808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.304393 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.304395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.304806 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.304807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.304859 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.305416 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:42.305421 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:42.307170 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:42.308731 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:42.308739 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:42.308741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:42.309053 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:42.309251 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:42.309252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:42.309252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:42.309329 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:42.309559 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:42.309567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:42.309646 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 22:53:42.309731 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 22:53:42.309888 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":44,"thread_start_us":99,"threads_started":1}
I20260928 22:53:42.312252 15508 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:38654:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 22:53:42.312525 15508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:53:42.313930 15786 tablet_service.cc:1467] Processing CreateTablet for tablet a103ad76a2ef415eb2ae017cbf26aa08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d3cae479c4834eeb823caee34df606c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:53:42.313995 15926 tablet_service.cc:1467] Processing CreateTablet for tablet a103ad76a2ef415eb2ae017cbf26aa08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d3cae479c4834eeb823caee34df606c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:53:42.314087 15786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a103ad76a2ef415eb2ae017cbf26aa08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:42.314039 15646 tablet_service.cc:1467] Processing CreateTablet for tablet a103ad76a2ef415eb2ae017cbf26aa08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d3cae479c4834eeb823caee34df606c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:53:42.314107 15926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a103ad76a2ef415eb2ae017cbf26aa08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:42.314191 15646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a103ad76a2ef415eb2ae017cbf26aa08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:42.315584 16017 tablet_bootstrap.cc:492] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a: Bootstrap starting.
I20260928 22:53:42.315734 16015 tablet_bootstrap.cc:492] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f: Bootstrap starting.
I20260928 22:53:42.316008 16017 tablet_bootstrap.cc:654] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:42.316147 16015 tablet_bootstrap.cc:654] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:42.316593 16016 tablet_bootstrap.cc:492] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82: Bootstrap starting.
I20260928 22:53:42.316865 16015 tablet_bootstrap.cc:492] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f: No bootstrap required, opened a new log
I20260928 22:53:42.316922 16015 ts_tablet_manager.cc:1397] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:42.316991 16016 tablet_bootstrap.cc:654] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:42.317085 16015 raft_consensus.cc:348] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.317165 16015 raft_consensus.cc:374] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:42.317188 16015 raft_consensus.cc:729] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a516ccc3a714f2c9a30e5100373683f, State: Initialized, Role: FOLLOWER
I20260928 22:53:42.317258 16015 consensus_queue.cc:260] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.317358 16015 ts_tablet_manager.cc:1428] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:42.317487 16023 tablet_replica.cc:439] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } }
I20260928 22:53:42.317564 16023 tablet_replica.cc:442] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:53:42.318296 16017 tablet_bootstrap.cc:492] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a: No bootstrap required, opened a new log
I20260928 22:53:42.318296 16016 tablet_bootstrap.cc:492] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82: No bootstrap required, opened a new log
I20260928 22:53:42.318351 16016 ts_tablet_manager.cc:1397] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:53:42.318351 16017 ts_tablet_manager.cc:1397] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 22:53:42.318493 16017 raft_consensus.cc:348] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.318492 16016 raft_consensus.cc:348] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.318572 16017 raft_consensus.cc:374] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:42.318572 16016 raft_consensus.cc:374] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:42.318595 16016 raft_consensus.cc:729] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28af8ebe1222484db27e886c54a7ea82, State: Initialized, Role: FOLLOWER
I20260928 22:53:42.318594 16017 raft_consensus.cc:729] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72fd149e81642c48814bc8e2f30c49a, State: Initialized, Role: FOLLOWER
I20260928 22:53:42.318640 16016 consensus_queue.cc:260] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.318645 16017 consensus_queue.cc:260] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.318730 16016 ts_tablet_manager.cc:1428] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 22:53:42.318730 16017 ts_tablet_manager.cc:1428] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:42.318792 16033 tablet_replica.cc:439] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } }
I20260928 22:53:42.318882 16033 tablet_replica.cc:442] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:53:42.318881 16022 tablet_replica.cc:439] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } }
I20260928 22:53:42.318970 16022 tablet_replica.cc:442] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:53:42.347087 16022 raft_consensus.cc:482] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:53:42.347189 16022 raft_consensus.cc:504] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.347536 16022 leader_election.cc:288] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a516ccc3a714f2c9a30e5100373683f (127.14.14.129:39879), e72fd149e81642c48814bc8e2f30c49a (127.14.14.130:34069)
I20260928 22:53:42.351037 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.347689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:39879 (local address 127.14.14.131:44911)
0928 22:53:42.347740 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:42.347753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:42.347795 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:42.347999 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:42.348000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:42.348003 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:42.348079 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.348080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.348822 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.348823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.349992 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.349995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.350770 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.350771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.350870 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.350874 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:42.350925 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:53:42.350964 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 22:53:42.351053 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.347737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:44911 (local address 127.14.14.129:39879)
0928 22:53:42.347790 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.347792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.347801 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:53:42.347911 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.347912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.347922 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.347972 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:42.348160 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.348724 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.350078 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.350695 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.350774 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.350783 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.350939 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.350979 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 22:53:42.351279 15993 tablet.cc:2369] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:53:42.351516 15666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a103ad76a2ef415eb2ae017cbf26aa08" candidate_uuid: "28af8ebe1222484db27e886c54a7ea82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a516ccc3a714f2c9a30e5100373683f" is_pre_election: true
I20260928 22:53:42.351593 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.347888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:51363 (local address 127.14.14.130:34069)
0928 22:53:42.347928 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.347930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.348066 (+ 136us) server_negotiation.cc:407] Connection header received
0928 22:53:42.348117 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.348118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.348127 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.348179 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:42.348319 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.349087 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.350711 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.351260 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.351323 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.351332 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.351497 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.351533 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:53:42.351606 15666 raft_consensus.cc:2454] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28af8ebe1222484db27e886c54a7ea82 in term 0.
I20260928 22:53:42.351788 15878 leader_election.cc:302] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28af8ebe1222484db27e886c54a7ea82, 7a516ccc3a714f2c9a30e5100373683f; no voters:
I20260928 22:53:42.351863 16056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.347707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:34069 (local address 127.14.14.131:51363)
0928 22:53:42.347996 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:42.348006 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:42.348064 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:42.348182 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:42.348183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:42.348185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:42.348250 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.348251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.349200 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.349201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.350629 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.350632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.351323 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.351324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.351406 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.351411 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:53:42.351696 (+ 285us) client_negotiation.cc:240] Negotiation successful
0928 22:53:42.351750 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":123,"threads_started":1}
I20260928 22:53:42.351917 16022 raft_consensus.cc:2781] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:53:42.351944 16022 raft_consensus.cc:482] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:53:42.351979 16022 raft_consensus.cc:3037] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:42.352873 16022 raft_consensus.cc:504] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.353000 16022 leader_election.cc:288] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [CANDIDATE]: Term 1 election: Requested vote from peers 7a516ccc3a714f2c9a30e5100373683f (127.14.14.129:39879), e72fd149e81642c48814bc8e2f30c49a (127.14.14.130:34069)
I20260928 22:53:42.353212 15666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a103ad76a2ef415eb2ae017cbf26aa08" candidate_uuid: "28af8ebe1222484db27e886c54a7ea82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a516ccc3a714f2c9a30e5100373683f"
I20260928 22:53:42.353315 15666 raft_consensus.cc:3037] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:42.353385 15806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a103ad76a2ef415eb2ae017cbf26aa08" candidate_uuid: "28af8ebe1222484db27e886c54a7ea82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72fd149e81642c48814bc8e2f30c49a" is_pre_election: true
I20260928 22:53:42.353459 15806 raft_consensus.cc:2454] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28af8ebe1222484db27e886c54a7ea82 in term 0.
I20260928 22:53:42.353505 15805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a103ad76a2ef415eb2ae017cbf26aa08" candidate_uuid: "28af8ebe1222484db27e886c54a7ea82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72fd149e81642c48814bc8e2f30c49a"
I20260928 22:53:42.353609 15805 raft_consensus.cc:3037] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:42.354074 15666 raft_consensus.cc:2454] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28af8ebe1222484db27e886c54a7ea82 in term 1.
I20260928 22:53:42.354242 15878 leader_election.cc:302] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28af8ebe1222484db27e886c54a7ea82, 7a516ccc3a714f2c9a30e5100373683f; no voters:
I20260928 22:53:42.354279 15805 raft_consensus.cc:2454] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28af8ebe1222484db27e886c54a7ea82 in term 1.
I20260928 22:53:42.354369 16022 raft_consensus.cc:2781] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:42.354749 16022 raft_consensus.cc:686] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [term 1 LEADER]: Becoming Leader. State: Replica: 28af8ebe1222484db27e886c54a7ea82, State: Running, Role: LEADER
I20260928 22:53:42.354847 16022 consensus_queue.cc:237] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } }
I20260928 22:53:42.355203 16057 tablet_replica.cc:439] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82: TxnStatusTablet state changed. Reason: New leader 28af8ebe1222484db27e886c54a7ea82. Latest consensus state: current_term: 1 leader_uuid: "28af8ebe1222484db27e886c54a7ea82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } }
I20260928 22:53:42.355345 16057 tablet_replica.cc:442] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:53:42.355571 16059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 22:53:42.355535 15508 catalog_manager.cc:5495] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28af8ebe1222484db27e886c54a7ea82 (127.14.14.131). New cstate: current_term: 1 leader_uuid: "28af8ebe1222484db27e886c54a7ea82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } health_report { overall_health: HEALTHY } } }
I20260928 22:53:42.355687 16059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 22:53:42.368176 16052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.365311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:41387 (local address 127.0.0.1:45328)
0928 22:53:42.365422 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:42.365436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:42.365500 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:42.365722 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:42.365723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:42.365724 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:53:42.365772 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.365774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.366682 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.366683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.367247 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.367248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.367700 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.367701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.367749 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.367763 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:53:42.368026 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:53:42.368032 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:53:42.368065 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 22:53:42.368104 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 22:53:42.368180 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.365434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.14.14.131:41387)
0928 22:53:42.365477 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.365479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.365488 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:53:42.365620 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.365621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.365640 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.365708 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:42.365832 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.366591 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.367325 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.367627 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.367696 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.367848 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:42.367966 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:42.368037 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.368079 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.368099 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:53:42.369273 15666 raft_consensus.cc:1261] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f [term 1 FOLLOWER]: Refusing update from remote peer 28af8ebe1222484db27e886c54a7ea82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:42.369305 15805 raft_consensus.cc:1261] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a [term 1 FOLLOWER]: Refusing update from remote peer 28af8ebe1222484db27e886c54a7ea82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:42.369606 16057 consensus_queue.cc:1035] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:42.369771 16022 consensus_queue.cc:1035] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:42.370405 16033 tablet_replica.cc:439] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a: TxnStatusTablet state changed. Reason: New leader 28af8ebe1222484db27e886c54a7ea82. Latest consensus state: current_term: 1 leader_uuid: "28af8ebe1222484db27e886c54a7ea82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } }
I20260928 22:53:42.370528 16033 tablet_replica.cc:442] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:53:42.370595 16023 tablet_replica.cc:439] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f: TxnStatusTablet state changed. Reason: New leader 28af8ebe1222484db27e886c54a7ea82. Latest consensus state: current_term: 1 leader_uuid: "28af8ebe1222484db27e886c54a7ea82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } }
I20260928 22:53:42.370705 16023 tablet_replica.cc:442] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:53:42.371209 16057 tablet_replica.cc:439] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28af8ebe1222484db27e886c54a7ea82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } }
I20260928 22:53:42.371299 16057 tablet_replica.cc:442] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:53:42.371438 16029 mvcc.cc:205] Tried to move back new op lower bound from 7334445147622576128 to 7334445147565998080. Current Snapshot: MvccSnapshot[applied={T|T < 7334445147622576128 or (T in {7334445147622576128})}]
I20260928 22:53:42.371948 16023 tablet_replica.cc:439] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28af8ebe1222484db27e886c54a7ea82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } }
I20260928 22:53:42.372028 16023 tablet_replica.cc:442] T a103ad76a2ef415eb2ae017cbf26aa08 P 7a516ccc3a714f2c9a30e5100373683f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:53:42.372731 16022 tablet_replica.cc:439] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28af8ebe1222484db27e886c54a7ea82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } }
I20260928 22:53:42.372823 16022 tablet_replica.cc:442] T a103ad76a2ef415eb2ae017cbf26aa08 P 28af8ebe1222484db27e886c54a7ea82: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:53:42.373067 16033 tablet_replica.cc:439] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28af8ebe1222484db27e886c54a7ea82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a516ccc3a714f2c9a30e5100373683f" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39879 } } peers { permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 } } peers { permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 } } }
I20260928 22:53:42.373148 16033 tablet_replica.cc:442] T a103ad76a2ef415eb2ae017cbf26aa08 P e72fd149e81642c48814bc8e2f30c49a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:53:42.375883 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.373074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.14.14.129:39879)
0928 22:53:42.373112 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.373114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.373132 (+ 18us) server_negotiation.cc:407] Connection header received
0928 22:53:42.373165 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.373166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.373185 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.373245 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:42.373520 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.374148 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.375200 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.375374 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.375422 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.375590 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:42.375687 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:42.375738 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.375805 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.375822 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:53:42.381392 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.378342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:54451 (local address 127.14.14.131:41387)
0928 22:53:42.378406 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.378408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.378524 (+ 116us) server_negotiation.cc:407] Connection header received
0928 22:53:42.378545 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.378546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.378556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.378605 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:42.378722 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.379481 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.380228 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.380541 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.380608 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.381132 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:42.381165 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:42.381224 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.381323 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.381342 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:53:42.381425 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.378274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:41387 (local address 127.14.14.129:54451)
0928 22:53:42.378428 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:42.378438 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:42.378490 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:42.378613 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:42.378613 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:42.378615 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:53:42.378669 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.378670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.379545 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.379546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.380108 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.380109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.380950 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.380951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.381018 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.381032 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:53:42.381240 (+ 208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:53:42.381242 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:53:42.381293 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:53:42.381332 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260928 22:53:42.383685 15946 raft_consensus.cc:1261] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82 [term 1 FOLLOWER]: Refusing update from remote peer 7a516ccc3a714f2c9a30e5100373683f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:42.383744 15805 raft_consensus.cc:1261] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a [term 1 FOLLOWER]: Refusing update from remote peer 7a516ccc3a714f2c9a30e5100373683f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:42.383925 16045 consensus_queue.cc:1035] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28af8ebe1222484db27e886c54a7ea82" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:42.384091 16023 consensus_queue.cc:1035] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72fd149e81642c48814bc8e2f30c49a" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:42.392139 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.389684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:35437 (local address 127.14.14.129:39879)
0928 22:53:42.389724 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.389726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.389822 (+ 96us) server_negotiation.cc:407] Connection header received
0928 22:53:42.389885 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.389886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.389893 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.389964 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:42.390110 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.390577 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.391361 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.391662 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.391747 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.391906 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:42.391940 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:42.391987 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.392046 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.392065 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:53:42.392160 15998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.389619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:39879 (local address 127.14.14.131:35437)
0928 22:53:42.389767 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:42.389779 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:42.389830 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:42.389971 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:42.389971 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:42.389973 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:53:42.390033 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.390035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.390685 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.390686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.391275 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:42.391276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.391747 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.391748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:42.391818 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.391836 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:53:42.391988 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:53:42.391990 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:53:42.392041 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:53:42.392083 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 22:53:42.401840 15508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:42.401960 15508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:42.402580 15508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f748720b50ef43ae92123676adb6cc0f: Sending DeleteTablet for 3 replicas of tablet 81c8ca158952470e862e5c30b6d5e920
I20260928 22:53:42.402840 15786 tablet_service.cc:1514] Processing DeleteTablet for tablet 81c8ca158952470e862e5c30b6d5e920 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:42 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:52640
I20260928 22:53:42.402957 16038 tablet_replica.cc:330] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a: stopping tablet replica
I20260928 22:53:42.402985 15646 tablet_service.cc:1514] Processing DeleteTablet for tablet 81c8ca158952470e862e5c30b6d5e920 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:42 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:39182
I20260928 22:53:42.403044 16038 raft_consensus.cc:2229] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:42.403093 16038 raft_consensus.cc:2258] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:42.403069 16040 tablet_replica.cc:330] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f: stopping tablet replica
I20260928 22:53:42.403157 16040 raft_consensus.cc:2229] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:53:42.403257 16040 raft_consensus.cc:2258] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:42.403445 16038 ts_tablet_manager.cc:1905] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:42.403574 16040 ts_tablet_manager.cc:1905] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:42.404942 15926 tablet_service.cc:1514] Processing DeleteTablet for tablet 81c8ca158952470e862e5c30b6d5e920 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:42 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:45312
I20260928 22:53:42.405050 16039 tablet_replica.cc:330] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82: stopping tablet replica
I20260928 22:53:42.405100 16039 raft_consensus.cc:2229] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:42.405153 16039 raft_consensus.cc:2258] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:42.405352 16038 ts_tablet_manager.cc:1918] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:53:42.405413 16038 log.cc:1196] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/81c8ca158952470e862e5c30b6d5e920
I20260928 22:53:42.405510 16039 ts_tablet_manager.cc:1905] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:42.405665 16038 ts_tablet_manager.cc:1939] T 81c8ca158952470e862e5c30b6d5e920 P e72fd149e81642c48814bc8e2f30c49a: Deleting consensus metadata
I20260928 22:53:42.405887 16040 ts_tablet_manager.cc:1918] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:53:42.405937 16040 log.cc:1196] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-0/wal/wals/81c8ca158952470e862e5c30b6d5e920
I20260928 22:53:42.406153 16040 ts_tablet_manager.cc:1939] T 81c8ca158952470e862e5c30b6d5e920 P 7a516ccc3a714f2c9a30e5100373683f: Deleting consensus metadata
I20260928 22:53:42.406445 15495 catalog_manager.cc:4841] TS e72fd149e81642c48814bc8e2f30c49a (127.14.14.130:34069): tablet 81c8ca158952470e862e5c30b6d5e920 (table test-table [id=c75740fb766a499e9e843705fb6309d8]) successfully deleted
I20260928 22:53:42.406531 15494 catalog_manager.cc:4841] TS 7a516ccc3a714f2c9a30e5100373683f (127.14.14.129:39879): tablet 81c8ca158952470e862e5c30b6d5e920 (table test-table [id=c75740fb766a499e9e843705fb6309d8]) successfully deleted
I20260928 22:53:42.407212 16039 ts_tablet_manager.cc:1918] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:53:42.407262 16039 log.cc:1196] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/81c8ca158952470e862e5c30b6d5e920
I20260928 22:53:42.407470 16039 ts_tablet_manager.cc:1939] T 81c8ca158952470e862e5c30b6d5e920 P 28af8ebe1222484db27e886c54a7ea82: Deleting consensus metadata
I20260928 22:53:42.407783 15496 catalog_manager.cc:4841] TS 28af8ebe1222484db27e886c54a7ea82 (127.14.14.131:41387): tablet 81c8ca158952470e862e5c30b6d5e920 (table test-table [id=c75740fb766a499e9e843705fb6309d8]) successfully deleted
Sep 28 22:53:42 dist-test-slave-kzfr krb5kdc[15464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636022, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.14.129@KRBTEST.COM
I20260928 22:53:42.418190 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.409849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.14.14.129:39879)
0928 22:53:42.409894 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.409897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.410045 (+ 148us) server_negotiation.cc:407] Connection header received
0928 22:53:42.410109 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.410111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.410121 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.410183 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:42.410770 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.411248 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.412106 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.412311 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.412384 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.415275 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:42.415284 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:42.415285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:42.415292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:42.416954 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:42.417329 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:42.417333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:42.417333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:42.417393 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:42.417613 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:42.417614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:42.417615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:42.417770 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:42.417947 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.417986 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.418075 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 22:53:42.418525 15706 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39210
I20260928 22:53:42.425735 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:42.419440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38670 (local address 127.14.14.190:33011)
0928 22:53:42.419485 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:42.419487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:42.419649 (+ 162us) server_negotiation.cc:407] Connection header received
0928 22:53:42.419719 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:42.419720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:42.419731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:42.419780 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:42.420264 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.420775 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.421547 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:42.421853 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:42.421922 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:42.423174 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:42.423176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:42.423176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:42.423183 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:42.424584 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:42.424951 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:42.424952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:42.424953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:42.425010 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:42.425218 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:42.425219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:42.425219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:42.425327 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:42.425430 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:42.425529 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 22:53:42.425632 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:53:42.426211 15508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38670
I20260928 22:53:42.426769 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 15581
I20260928 22:53:42.431903 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 15721
I20260928 22:53:42.438968 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 15861
I20260928 22:53:42.443670 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 15477
2026-09-28T22:53:42Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2134 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:42 dist-test-slave-kzfr krb5kdc[16080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:42 dist-test-slave-kzfr krb5kdc[16080](info): set up 2 sockets
Sep 28 22:53:42 dist-test-slave-kzfr krb5kdc[16080](info): commencing operation
krb5kdc: starting...
W20260928 22:53:43.481619 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:53:43 dist-test-slave-kzfr krb5kdc[16080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:53:43.601280 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:38387
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:38387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:43.682088 16093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:43.682291 16093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:43.682312 16093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:43.683668 16093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:43.683708 16093 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 22:53:43.683722 16093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:43.683735 16093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:43.683748 16093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:43.683760 16093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:43.685225 16093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:38387
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:38387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:43.685473 16093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:43.685758 16093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:43.690869 16098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:43.690918 16099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:43.691022 16101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:43.691041 16093 server_base.cc:1034] running on GCE node
I20260928 22:53:43.691380 16093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:43.691690 16093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:43.692865 16093 hybrid_clock.cc:648] HybridClock initialized: now 1790636023692828 us; error 42 us; skew 500 ppm
Sep 28 22:53:43 dist-test-slave-kzfr krb5kdc[16080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636023, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:43.695464 16093 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:53:43.695978 16093 webserver.cc:460] Webserver started at http://127.14.14.190:37877/ using document root <none> and password file <none>
I20260928 22:53:43.696220 16093 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:43.696291 16093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:43.696442 16093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:43.697490 16093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "d88d3c8c71134d74a8c0e6c4ffb5d3bc"
format_stamp: "Formatted at 2026-09-28 22:53:43 on dist-test-slave-kzfr"
server_key: "eaa94f7d79110ad2f161b737802b383e"
server_key_iv: "858b68738cc46878cb81076e9fc876c5"
server_key_version: "encryptionkey@0"
I20260928 22:53:43.697925 16093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "d88d3c8c71134d74a8c0e6c4ffb5d3bc"
format_stamp: "Formatted at 2026-09-28 22:53:43 on dist-test-slave-kzfr"
server_key: "eaa94f7d79110ad2f161b737802b383e"
server_key_iv: "858b68738cc46878cb81076e9fc876c5"
server_key_version: "encryptionkey@0"
I20260928 22:53:43.699229 16093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:43.699975 16108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:43.700161 16093 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:43.700253 16093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "d88d3c8c71134d74a8c0e6c4ffb5d3bc"
format_stamp: "Formatted at 2026-09-28 22:53:43 on dist-test-slave-kzfr"
server_key: "eaa94f7d79110ad2f161b737802b383e"
server_key_iv: "858b68738cc46878cb81076e9fc876c5"
server_key_version: "encryptionkey@0"
I20260928 22:53:43.700333 16093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:43.712102 16093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:43.714427 16093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:43.714591 16093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:43.719334 16093 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:38387
I20260928 22:53:43.719378 16170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:38387 every 8 connection(s)
I20260928 22:53:43.719740 16093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:43.720353 16171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:43.722702 16171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc: Bootstrap starting.
I20260928 22:53:43.723304 16171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:43.723605 16171 log.cc:824] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:43.724285 16171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc: No bootstrap required, opened a new log
I20260928 22:53:43.726240 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 16093
I20260928 22:53:43.726353 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:53:43.726521 14394 external_mini_cluster.cc:1428] Setting key c0836557533b20f8db4b9d1daa011214
I20260928 22:53:43.726737 16171 raft_consensus.cc:348] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88d3c8c71134d74a8c0e6c4ffb5d3bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38387 } }
I20260928 22:53:43.726907 16171 raft_consensus.cc:374] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:43.726945 16171 raft_consensus.cc:729] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88d3c8c71134d74a8c0e6c4ffb5d3bc, State: Initialized, Role: FOLLOWER
I20260928 22:53:43.727074 16171 consensus_queue.cc:260] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88d3c8c71134d74a8c0e6c4ffb5d3bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38387 } }
I20260928 22:53:43.727169 16171 raft_consensus.cc:388] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:43.727214 16171 raft_consensus.cc:482] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:43.727259 16171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:43.727779 16171 raft_consensus.cc:504] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88d3c8c71134d74a8c0e6c4ffb5d3bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38387 } }
I20260928 22:53:43.727900 16171 leader_election.cc:302] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d88d3c8c71134d74a8c0e6c4ffb5d3bc; no voters:
I20260928 22:53:43.728070 16171 leader_election.cc:288] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:43.728130 16176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:43.728277 16176 raft_consensus.cc:686] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [term 1 LEADER]: Becoming Leader. State: Replica: d88d3c8c71134d74a8c0e6c4ffb5d3bc, State: Running, Role: LEADER
I20260928 22:53:43.728333 16171 sys_catalog.cc:564] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:43.728368 16176 consensus_queue.cc:237] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88d3c8c71134d74a8c0e6c4ffb5d3bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38387 } }
I20260928 22:53:43.728701 16176 sys_catalog.cc:455] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d88d3c8c71134d74a8c0e6c4ffb5d3bc. Latest consensus state: current_term: 1 leader_uuid: "d88d3c8c71134d74a8c0e6c4ffb5d3bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88d3c8c71134d74a8c0e6c4ffb5d3bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38387 } } }
I20260928 22:53:43.728778 16176 sys_catalog.cc:458] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:43.728714 16177 sys_catalog.cc:455] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d88d3c8c71134d74a8c0e6c4ffb5d3bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88d3c8c71134d74a8c0e6c4ffb5d3bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38387 } } }
I20260928 22:53:43.729025 16177 sys_catalog.cc:458] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:43.729830 16182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:43.730479 16182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:43.732249 16182 catalog_manager.cc:1347] Generated new cluster ID: 2fa6d4ef2daa4d8083eb1aa4476704cb
I20260928 22:53:43.732301 16182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:53:43 dist-test-slave-kzfr krb5kdc[16080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:43.737485 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:43.727018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.14.14.190:38387)
0928 22:53:43.727186 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:43.727196 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:43.727230 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:53:43.727422 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:43.727427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:43.727469 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:43.727605 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:43.728363 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:43.730025 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:43.730902 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:43.731529 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:43.731646 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:43.734227 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:43.734231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:43.734231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:43.734243 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:43.735962 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:43.736347 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:43.736350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:43.736351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:43.736461 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:43.736678 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:43.736679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:43.736680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:43.736907 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:43.737078 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:43.737185 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 22:53:43.737325 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260928 22:53:43.741451 16182 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:43.742179 16182 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:43.745929 16182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc: Generated new TSK 0
I20260928 22:53:43.746096 16182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:53:43.780567 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38387
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:43.885192 16197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:43.885419 16197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:43.885460 16197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:43.886664 16197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:43.886731 16197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:43.886765 16197 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 22:53:43.886791 16197 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:53:43.886827 16197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:53:43.888309 16197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.14.190:38387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:43.888582 16197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:43.888809 16197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:43.895061 16202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:43.895264 16205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:43.895222 16197 server_base.cc:1034] running on GCE node
W20260928 22:53:43.895061 16203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:43.895529 16197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:43.895843 16197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:43.897017 16197 hybrid_clock.cc:648] HybridClock initialized: now 1790636023897003 us; error 26 us; skew 500 ppm
Sep 28 22:53:43 dist-test-slave-kzfr krb5kdc[16080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636023, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:43.899901 16197 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:53:43.900446 16197 webserver.cc:460] Webserver started at http://127.14.14.129:36235/ using document root <none> and password file <none>
I20260928 22:53:43.900725 16197 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:43.900799 16197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:43.900962 16197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:43.902123 16197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "d4fff65288a54c3fb142b3c7fdea7d7c"
format_stamp: "Formatted at 2026-09-28 22:53:43 on dist-test-slave-kzfr"
server_key: "409707a27c77305379513c1cd2196fb1"
server_key_iv: "dd1431e74395352657c5886525dba680"
server_key_version: "encryptionkey@0"
I20260928 22:53:43.902504 16197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "d4fff65288a54c3fb142b3c7fdea7d7c"
format_stamp: "Formatted at 2026-09-28 22:53:43 on dist-test-slave-kzfr"
server_key: "409707a27c77305379513c1cd2196fb1"
server_key_iv: "dd1431e74395352657c5886525dba680"
server_key_version: "encryptionkey@0"
I20260928 22:53:43.903944 16197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:53:43.904726 16212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:43.904937 16197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:43.905045 16197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "d4fff65288a54c3fb142b3c7fdea7d7c"
format_stamp: "Formatted at 2026-09-28 22:53:43 on dist-test-slave-kzfr"
server_key: "409707a27c77305379513c1cd2196fb1"
server_key_iv: "dd1431e74395352657c5886525dba680"
server_key_version: "encryptionkey@0"
I20260928 22:53:43.905118 16197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:43.916186 16197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:43.918496 16197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:43.918659 16197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:43.919145 16197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:43.919214 16197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:43.919257 16197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:43.919301 16197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:53:43 dist-test-slave-kzfr krb5kdc[16080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636023, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:43.926410 16197 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:36883
I20260928 22:53:43.926632 16327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:36883 every 8 connection(s)
I20260928 22:53:43.926810 16197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:53:43.928471 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 16197
I20260928 22:53:43.928579 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:53:43.928784 14394 external_mini_cluster.cc:1428] Setting key 6abd2d88565d1a79537b1636f833459b
I20260928 22:53:43.931371 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:43.919668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:51379 (local address 127.14.14.190:38387)
0928 22:53:43.919709 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:43.919711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:43.919949 (+ 238us) server_negotiation.cc:407] Connection header received
0928 22:53:43.920490 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:43.920492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:43.920505 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:43.920556 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:43.921399 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:43.922288 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:43.923555 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:43.923913 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:43.923995 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:43.926803 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:43.926805 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:43.926806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:43.926813 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:43.928447 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:43.929111 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:43.929112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:43.929113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:43.929174 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:43.930715 (+ 1541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:43.930716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:43.930716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:43.930827 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:43.930919 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:43.931196 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 22:53:43.931281 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:53:43.931584 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:43.919661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38387 (local address 127.14.14.129:51379)
0928 22:53:43.919829 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:43.919862 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:43.920363 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:43.920635 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:43.920638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:43.921002 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:43.921276 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:43.921283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:43.922389 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:43.922391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:43.923447 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:43.923449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:43.924019 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:43.924020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:43.924181 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:43.924728 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:43.924741 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:43.926695 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:43.928661 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:43.928668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:43.928670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:43.929001 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:43.929260 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:43.929261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:43.929261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:43.929409 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:43.930912 (+ 1503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:43.930918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:43.931059 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 22:53:43.931173 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 22:53:43.931337 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260928 22:53:43.932117 16328 heartbeater.cc:347] Connected to a master server at 127.14.14.190:38387
I20260928 22:53:43.932202 16328 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:43.932377 16328 heartbeater.cc:511] Master 127.14.14.190:38387 requested a full tablet report, sending...
I20260928 22:53:43.932840 16124 ts_manager.cc:194] Registered new tserver with Master: d4fff65288a54c3fb142b3c7fdea7d7c (127.14.14.129:36883)
I20260928 22:53:43.933672 16124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:51379
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
I20260928 22:53:43.938985 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:43.933258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38387 (local address 127.14.14.129:42577)
0928 22:53:43.933429 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:43.933441 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:43.933500 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:43.933702 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:43.933704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:43.933984 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:43.934045 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:43.934047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:43.934854 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:43.934854 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:43.935362 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:43.935363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:43.935748 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:43.935749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:43.935822 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:43.936291 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:43.936295 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:43.936580 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:43.938091 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:43.938092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:43.938093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:43.938272 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:43.938450 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:43.938451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:43.938451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:43.938501 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:43.938705 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:43.938705 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:43.938740 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 22:53:43.938794 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 22:53:43.938907 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
I20260928 22:53:43.938990 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:43.933315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:42577 (local address 127.14.14.190:38387)
0928 22:53:43.933348 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:43.933349 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:43.933504 (+ 155us) server_negotiation.cc:407] Connection header received
0928 22:53:43.933557 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:43.933558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:43.933567 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:43.933645 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:43.934114 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:43.934751 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:43.935478 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:43.935675 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:43.935750 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:43.936688 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:43.936689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:43.936689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:43.936696 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:43.938011 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:43.938345 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:43.938345 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:43.938346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:43.938393 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:43.938558 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:43.938559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:43.938559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:43.938650 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:43.938699 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:43.938814 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 22:53:43.938873 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:53:43.956822 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38387
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:44.038743 16337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:44.038941 16337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:44.038965 16337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:44.040108 16337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:44.040151 16337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:44.040172 16337 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 22:53:44.040184 16337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:53:44.040246 16337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:53:44.041711 16337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.14.190:38387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:44.041949 16337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:44.042194 16337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:44.047281 16343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:44.047309 16342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:44.047344 16337 server_base.cc:1034] running on GCE node
W20260928 22:53:44.047302 16345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:44.047631 16337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:44.047891 16337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:44.049062 16337 hybrid_clock.cc:648] HybridClock initialized: now 1790636024049041 us; error 41 us; skew 500 ppm
Sep 28 22:53:44 dist-test-slave-kzfr krb5kdc[16080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636024, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:44.051537 16337 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:53:44.051957 16337 webserver.cc:460] Webserver started at http://127.14.14.130:37845/ using document root <none> and password file <none>
I20260928 22:53:44.052196 16337 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:44.052263 16337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:44.052413 16337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:44.053417 16337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "1535b5448d064ccda6f5ff6f99e10d9f"
format_stamp: "Formatted at 2026-09-28 22:53:44 on dist-test-slave-kzfr"
server_key: "bbc4dcbdf186a384b7d04aa061d28d05"
server_key_iv: "db989227b8425453a902df1a4253e0c7"
server_key_version: "encryptionkey@0"
I20260928 22:53:44.053798 16337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "1535b5448d064ccda6f5ff6f99e10d9f"
format_stamp: "Formatted at 2026-09-28 22:53:44 on dist-test-slave-kzfr"
server_key: "bbc4dcbdf186a384b7d04aa061d28d05"
server_key_iv: "db989227b8425453a902df1a4253e0c7"
server_key_version: "encryptionkey@0"
I20260928 22:53:44.055084 16337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:53:44.055785 16352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:44.055948 16337 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:53:44.056048 16337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "1535b5448d064ccda6f5ff6f99e10d9f"
format_stamp: "Formatted at 2026-09-28 22:53:44 on dist-test-slave-kzfr"
server_key: "bbc4dcbdf186a384b7d04aa061d28d05"
server_key_iv: "db989227b8425453a902df1a4253e0c7"
server_key_version: "encryptionkey@0"
I20260928 22:53:44.056129 16337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:44.068153 16337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:44.069895 16337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:44.070043 16337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:44.070521 16337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:44.070585 16337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:44.070652 16337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:44.070683 16337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:53:44 dist-test-slave-kzfr krb5kdc[16080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636024, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:44.078114 16337 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:45111
I20260928 22:53:44.078154 16467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:45111 every 8 connection(s)
I20260928 22:53:44.078465 16337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:53:44.080848 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.070981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:57517 (local address 127.14.14.190:38387)
0928 22:53:44.071058 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:44.071060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:44.071315 (+ 255us) server_negotiation.cc:407] Connection header received
0928 22:53:44.072066 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:44.072068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:44.072081 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:44.072138 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:44.073055 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.073936 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.075601 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.075941 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.076004 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.078281 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:44.078283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:44.078283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:44.078290 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:44.079723 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:44.080088 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:44.080089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:44.080089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:44.080137 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:44.080326 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:44.080327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:44.080327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:44.080424 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:44.080482 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:44.080723 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 22:53:44.080786 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:53:44.080996 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.070960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38387 (local address 127.14.14.130:57517)
0928 22:53:44.071157 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:44.071208 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:44.071948 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:44.072231 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:44.072235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:44.072633 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:44.072949 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.072956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.074038 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.074039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.075492 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.075495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.076025 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.076026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.076190 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.076738 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:44.076747 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:44.078158 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:44.079799 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:44.079804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:44.079806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:44.080021 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:44.080194 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:44.080194 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:44.080194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:44.080276 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:44.080499 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:44.080503 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:44.080607 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 22:53:44.080693 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 22:53:44.080806 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
I20260928 22:53:44.081451 16468 heartbeater.cc:347] Connected to a master server at 127.14.14.190:38387
I20260928 22:53:44.081558 16468 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:44.081664 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 16337
I20260928 22:53:44.081751 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:53:44.081782 16468 heartbeater.cc:511] Master 127.14.14.190:38387 requested a full tablet report, sending...
I20260928 22:53:44.081947 14394 external_mini_cluster.cc:1428] Setting key 91eef697dbac89ae9dfa608a4bf8a72f
I20260928 22:53:44.082183 16125 ts_manager.cc:194] Registered new tserver with Master: 1535b5448d064ccda6f5ff6f99e10d9f (127.14.14.130:45111)
I20260928 22:53:44.082651 16125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:57517
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
I20260928 22:53:44.090461 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.082531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38387 (local address 127.14.14.130:57611)
0928 22:53:44.082651 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:44.082661 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:44.082730 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:44.084253 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:44.084254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:44.084398 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:44.084447 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.084448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.085304 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.085305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.086238 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.086241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.086572 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.086574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.086667 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.087048 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:44.087053 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:44.087473 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:44.089202 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:44.089203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:44.089204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:44.089416 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:44.089722 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:44.089723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:44.089724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:44.089790 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:44.090102 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:44.090103 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:44.090149 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 22:53:44.090223 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 22:53:44.090319 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
I20260928 22:53:44.090575 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.084057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:57611 (local address 127.14.14.190:38387)
0928 22:53:44.084130 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:44.084132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:44.084142 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:53:44.084163 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:44.084164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:44.084175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:44.084231 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:44.084517 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.085203 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.086318 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.086495 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.086550 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.087565 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:44.087566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:44.087566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:44.087572 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:44.089115 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:44.089549 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:44.089550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:44.089550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:44.089656 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:44.089869 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:44.089869 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:44.089870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:44.090028 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:44.090085 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:44.090385 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 22:53:44.090472 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:53:44.109550 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38387
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:44.190930 16477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:44.191156 16477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:44.191180 16477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:44.192370 16477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:44.192415 16477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:44.192436 16477 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 22:53:44.192449 16477 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:53:44.192466 16477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:53:44.194026 16477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.14.190:38387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:44.194217 16477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:44.194394 16477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:53:44.199900 16477 server_base.cc:1034] running on GCE node
W20260928 22:53:44.199815 16482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:44.199815 16483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:44.199819 16485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:44.200294 16477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:44.200575 16477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:44.201735 16477 hybrid_clock.cc:648] HybridClock initialized: now 1790636024201703 us; error 41 us; skew 500 ppm
Sep 28 22:53:44 dist-test-slave-kzfr krb5kdc[16080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636024, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:44.204422 16477 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:53:44.204867 16477 webserver.cc:460] Webserver started at http://127.14.14.131:37577/ using document root <none> and password file <none>
I20260928 22:53:44.205099 16477 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:44.205168 16477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:44.205313 16477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:44.206451 16477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "0a69b854c27145c19fe37aaacfda117e"
format_stamp: "Formatted at 2026-09-28 22:53:44 on dist-test-slave-kzfr"
server_key: "b0f6dc90b11b1b5f9d59d07c3f0fa80b"
server_key_iv: "42eb75dfc08238c1f239981f0c6716d1"
server_key_version: "encryptionkey@0"
I20260928 22:53:44.206822 16477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "0a69b854c27145c19fe37aaacfda117e"
format_stamp: "Formatted at 2026-09-28 22:53:44 on dist-test-slave-kzfr"
server_key: "b0f6dc90b11b1b5f9d59d07c3f0fa80b"
server_key_iv: "42eb75dfc08238c1f239981f0c6716d1"
server_key_version: "encryptionkey@0"
I20260928 22:53:44.208266 16477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:53:44.209048 16492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:44.209246 16477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:53:44.209343 16477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "0a69b854c27145c19fe37aaacfda117e"
format_stamp: "Formatted at 2026-09-28 22:53:44 on dist-test-slave-kzfr"
server_key: "b0f6dc90b11b1b5f9d59d07c3f0fa80b"
server_key_iv: "42eb75dfc08238c1f239981f0c6716d1"
server_key_version: "encryptionkey@0"
I20260928 22:53:44.209434 16477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:44.222573 16477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:44.224965 16477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:44.225258 16477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:44.225966 16477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:44.226024 16477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:44.226063 16477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:44.226085 16477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:53:44 dist-test-slave-kzfr krb5kdc[16080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636024, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:44.233608 16477 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:40717
I20260928 22:53:44.233671 16607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:40717 every 8 connection(s)
I20260928 22:53:44.234006 16477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:53:44.234328 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 16477
I20260928 22:53:44.234436 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:53:44.234699 14394 external_mini_cluster.cc:1428] Setting key 9adcf6ba9b313175b773fa5615258221
I20260928 22:53:44.236344 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.226225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:48109 (local address 127.14.14.190:38387)
0928 22:53:44.226272 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:44.226274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:44.226476 (+ 202us) server_negotiation.cc:407] Connection header received
0928 22:53:44.226958 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:44.226959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:44.226973 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:44.227050 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:44.227626 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.228448 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.229842 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.230174 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.230261 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.233763 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:44.233765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:44.233765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:44.233774 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:44.235299 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:44.235685 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:44.235686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:44.235686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:44.235738 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:44.235897 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:44.235897 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:44.235897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:44.235989 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:44.236042 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:44.236228 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 22:53:44.236284 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:53:44.236508 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.226222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38387 (local address 127.14.14.131:48109)
0928 22:53:44.226373 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:44.226406 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:44.226858 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:44.227113 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:44.227116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:44.227351 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:44.227528 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.227531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.228542 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.228543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.229741 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.229744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.230272 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.230274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.230448 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.231019 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:44.231030 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:44.233676 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:44.235375 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:44.235381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:44.235383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:44.235619 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:44.235781 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:44.235782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:44.235782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:44.235862 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:44.236035 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:44.236038 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:44.236136 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 22:53:44.236212 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 22:53:44.236319 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260928 22:53:44.236984 16608 heartbeater.cc:347] Connected to a master server at 127.14.14.190:38387
I20260928 22:53:44.237064 16608 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:44.237236 16608 heartbeater.cc:511] Master 127.14.14.190:38387 requested a full tablet report, sending...
I20260928 22:53:44.237564 16125 ts_manager.cc:194] Registered new tserver with Master: 0a69b854c27145c19fe37aaacfda117e (127.14.14.131:40717)
I20260928 22:53:44.238020 16125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:48109
I20260928 22:53:44.243414 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.238079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:52773 (local address 127.14.14.190:38387)
0928 22:53:44.238118 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:44.238120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:44.238236 (+ 116us) server_negotiation.cc:407] Connection header received
0928 22:53:44.238322 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:44.238323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:44.238333 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:44.238393 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:44.238676 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.239185 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.239835 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.240008 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.240061 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.240929 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:44.240931 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:44.240931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:44.240937 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:44.242121 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:44.242528 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:44.242544 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:44.242544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:44.242594 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:44.242842 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:44.242842 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:44.242843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:44.242980 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:44.243044 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:44.243254 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 22:53:44.243333 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:53:44.243480 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.238030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38387 (local address 127.14.14.131:52773)
0928 22:53:44.238180 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:44.238190 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:44.238253 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:44.238396 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:44.238396 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:44.238541 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:44.238606 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.238607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.239272 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.239273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.239763 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.239764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.240072 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.240073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.240124 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.240559 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:44.240564 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:44.240832 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:44.242190 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:44.242191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:44.242191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:44.242426 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:44.242683 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:44.242684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:44.242684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:44.242769 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:44.243056 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:44.243057 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:44.243120 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 22:53:44.243209 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 22:53:44.243331 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
I20260928 22:53:44.246181 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 22:53:44 dist-test-slave-kzfr krb5kdc[16080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 22:53:44 dist-test-slave-kzfr krb5kdc[16080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:44.267971 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.261641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.14.14.190:38387)
0928 22:53:44.261696 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:44.261697 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:44.261707 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:53:44.261740 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:44.261741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:44.261749 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:44.261798 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:44.262330 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.262807 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.263500 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.263677 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.263724 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.265934 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:44.265935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:44.265935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:44.265942 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:44.267116 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:44.267413 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:44.267414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:44.267414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:44.267463 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:44.267606 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:44.267606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:44.267606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:44.267698 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:44.267770 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:44.267853 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 22:53:44.267908 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:53:44.269680 16125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:53:44.271318 16125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:53:44.278237 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.274074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.14.14.131:40717)
0928 22:53:44.274271 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:44.274276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:44.274615 (+ 339us) server_negotiation.cc:407] Connection header received
0928 22:53:44.274727 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:44.274729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:44.274744 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:44.274817 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:44.274923 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.275588 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.276908 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.277707 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.277777 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.277837 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:44.278075 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 22:53:44.278119 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
I20260928 22:53:44.278270 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.273666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:45111 (local address 127.0.0.1:37766)
0928 22:53:44.273817 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:44.273861 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:44.273938 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:44.274236 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:44.274239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:44.274242 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:44.274323 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.274325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.275712 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.275713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.277001 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.277004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.277995 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.277996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.278057 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.278060 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:53:44.278105 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 22:53:44.278141 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":34,"thread_start_us":86,"threads_started":1}
I20260928 22:53:44.278366 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.273844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.14.14.130:45111)
0928 22:53:44.274038 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:44.274043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:44.274054 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:53:44.274097 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:44.274099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:44.274116 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:44.274220 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:44.274419 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.275606 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.277087 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.277851 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.277914 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.277964 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:44.278209 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 22:53:44.278252 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":83,"threads_started":1}
I20260928 22:53:44.278281 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.273668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:36883 (local address 127.0.0.1:40070)
0928 22:53:44.273994 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:44.274003 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:44.274046 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:44.274722 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:44.274724 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:44.274727 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:44.274769 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.274771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.276054 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.276055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.277139 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.277141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.277901 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.277902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.277978 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.277982 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:44.278053 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 22:53:44.278109 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":2,"thread_start_us":46,"threads_started":1}
I20260928 22:53:44.278805 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.273880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:40717 (local address 127.0.0.1:55972)
0928 22:53:44.274536 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:44.274546 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:44.274620 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:44.274816 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:44.274817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:44.274819 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:44.274867 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.274869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.275709 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.275710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.276811 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.276813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.277875 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.277876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.277955 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.277970 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 22:53:44.278042 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 22:53:44.278107 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":52,"thread_start_us":62,"threads_started":1}
I20260928 22:53:44.279027 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.273827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.14.14.129:36883)
0928 22:53:44.274346 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:44.274351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:44.274380 (+ 29us) server_negotiation.cc:407] Connection header received
0928 22:53:44.274418 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:44.274420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:44.274436 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:44.274722 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:44.274859 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.275959 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.277228 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.277802 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.277898 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.277952 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:44.278171 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 22:53:44.278929 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":86,"threads_started":1}
I20260928 22:53:44.279206 16402 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9a3948859a48e78a20f8d9d8e7c714 (DEFAULT_TABLE table=test-table [id=f652a5032deb48ba82d164a1b6758b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:44.279445 16542 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9a3948859a48e78a20f8d9d8e7c714 (DEFAULT_TABLE table=test-table [id=f652a5032deb48ba82d164a1b6758b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:44.279562 16402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9a3948859a48e78a20f8d9d8e7c714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:44.279559 16262 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9a3948859a48e78a20f8d9d8e7c714 (DEFAULT_TABLE table=test-table [id=f652a5032deb48ba82d164a1b6758b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:44.279737 16542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9a3948859a48e78a20f8d9d8e7c714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:44.279801 16262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9a3948859a48e78a20f8d9d8e7c714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:44.282742 16631 tablet_bootstrap.cc:492] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c: Bootstrap starting.
I20260928 22:53:44.283223 16633 tablet_bootstrap.cc:492] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f: Bootstrap starting.
I20260928 22:53:44.283308 16632 tablet_bootstrap.cc:492] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e: Bootstrap starting.
I20260928 22:53:44.283416 16631 tablet_bootstrap.cc:654] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:44.283761 16631 log.cc:824] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:44.283874 16633 tablet_bootstrap.cc:654] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:44.284130 16632 tablet_bootstrap.cc:654] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:44.284173 16633 log.cc:824] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:44.284466 16632 log.cc:824] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:44.284596 16631 tablet_bootstrap.cc:492] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c: No bootstrap required, opened a new log
I20260928 22:53:44.284670 16631 ts_tablet_manager.cc:1397] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:53:44.284930 16633 tablet_bootstrap.cc:492] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f: No bootstrap required, opened a new log
I20260928 22:53:44.284992 16633 ts_tablet_manager.cc:1397] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:53:44.285220 16632 tablet_bootstrap.cc:492] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e: No bootstrap required, opened a new log
I20260928 22:53:44.285276 16632 ts_tablet_manager.cc:1397] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:53:44.286518 16633 raft_consensus.cc:348] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.286633 16633 raft_consensus.cc:374] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:44.286664 16633 raft_consensus.cc:729] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1535b5448d064ccda6f5ff6f99e10d9f, State: Initialized, Role: FOLLOWER
I20260928 22:53:44.286782 16633 consensus_queue.cc:260] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.286743 16631 raft_consensus.cc:348] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.286890 16631 raft_consensus.cc:374] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:44.286922 16631 raft_consensus.cc:729] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4fff65288a54c3fb142b3c7fdea7d7c, State: Initialized, Role: FOLLOWER
I20260928 22:53:44.287045 16633 ts_tablet_manager.cc:1428] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:53:44.287060 16468 heartbeater.cc:503] Master 127.14.14.190:38387 was elected leader, sending a full tablet report...
I20260928 22:53:44.287055 16631 consensus_queue.cc:260] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.287314 16631 ts_tablet_manager.cc:1428] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:53:44.287354 16328 heartbeater.cc:503] Master 127.14.14.190:38387 was elected leader, sending a full tablet report...
I20260928 22:53:44.287277 16632 raft_consensus.cc:348] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.287441 16632 raft_consensus.cc:374] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:44.287477 16632 raft_consensus.cc:729] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69b854c27145c19fe37aaacfda117e, State: Initialized, Role: FOLLOWER
I20260928 22:53:44.287578 16632 consensus_queue.cc:260] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.287827 16632 ts_tablet_manager.cc:1428] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:53:44.287845 16608 heartbeater.cc:503] Master 127.14.14.190:38387 was elected leader, sending a full tablet report...
W20260928 22:53:44.329159 16469 tablet.cc:2369] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:53:44.427795 16329 tablet.cc:2369] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:53:44.484633 16609 tablet.cc:2369] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:53:44.499482 16639 raft_consensus.cc:482] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:53:44.499624 16639 raft_consensus.cc:504] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.500085 16639 leader_election.cc:288] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4fff65288a54c3fb142b3c7fdea7d7c (127.14.14.129:36883), 1535b5448d064ccda6f5ff6f99e10d9f (127.14.14.130:45111)
I20260928 22:53:44.503508 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.500447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:41797 (local address 127.14.14.129:36883)
0928 22:53:44.500599 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:44.500601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:44.500739 (+ 138us) server_negotiation.cc:407] Connection header received
0928 22:53:44.500765 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:44.500766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:44.500775 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:44.500829 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:44.501062 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.501567 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.502577 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.503123 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.503208 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.503216 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:44.503386 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 22:53:44.503425 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260928 22:53:44.503535 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.500451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:43279 (local address 127.14.14.130:45111)
0928 22:53:44.500497 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:44.500499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:44.500512 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:53:44.500627 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:44.500629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:44.500645 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:44.500708 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:44.500906 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.501563 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.502590 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.503175 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.503260 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.503268 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:44.503430 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 22:53:44.503466 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:53:44.503539 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.500350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:45111 (local address 127.14.14.131:43279)
0928 22:53:44.500430 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:44.500461 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:44.500512 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:44.500740 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:44.500741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:44.500743 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:44.500826 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.500828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.501710 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.501711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.502490 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.502492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.503257 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.503258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.503344 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.503347 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:53:44.503422 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 22:53:44.503462 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 22:53:44.503539 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.500350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:36883 (local address 127.14.14.131:41797)
0928 22:53:44.500601 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:44.500612 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:44.500695 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:44.500854 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:44.500855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:44.500856 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:44.500977 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.500979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.501720 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.501721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.502480 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.502482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.503206 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.503207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.503302 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.503306 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:44.503381 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 22:53:44.503420 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":45,"thread_start_us":127,"threads_started":1}
I20260928 22:53:44.503903 16422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9a3948859a48e78a20f8d9d8e7c714" candidate_uuid: "0a69b854c27145c19fe37aaacfda117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" is_pre_election: true
I20260928 22:53:44.504055 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9a3948859a48e78a20f8d9d8e7c714" candidate_uuid: "0a69b854c27145c19fe37aaacfda117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" is_pre_election: true
I20260928 22:53:44.504103 16422 raft_consensus.cc:2454] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a69b854c27145c19fe37aaacfda117e in term 0.
I20260928 22:53:44.504192 16265 raft_consensus.cc:2454] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a69b854c27145c19fe37aaacfda117e in term 0.
I20260928 22:53:44.504384 16493 leader_election.cc:302] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a69b854c27145c19fe37aaacfda117e, 1535b5448d064ccda6f5ff6f99e10d9f, d4fff65288a54c3fb142b3c7fdea7d7c; no voters:
I20260928 22:53:44.504539 16639 raft_consensus.cc:2781] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:53:44.504599 16639 raft_consensus.cc:482] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:53:44.504623 16639 raft_consensus.cc:3037] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:44.505540 16639 raft_consensus.cc:504] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.505710 16639 leader_election.cc:288] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [CANDIDATE]: Term 1 election: Requested vote from peers d4fff65288a54c3fb142b3c7fdea7d7c (127.14.14.129:36883), 1535b5448d064ccda6f5ff6f99e10d9f (127.14.14.130:45111)
I20260928 22:53:44.505800 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9a3948859a48e78a20f8d9d8e7c714" candidate_uuid: "0a69b854c27145c19fe37aaacfda117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c"
I20260928 22:53:44.505887 16422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9a3948859a48e78a20f8d9d8e7c714" candidate_uuid: "0a69b854c27145c19fe37aaacfda117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1535b5448d064ccda6f5ff6f99e10d9f"
I20260928 22:53:44.505888 16265 raft_consensus.cc:3037] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:44.505960 16422 raft_consensus.cc:3037] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:44.506783 16265 raft_consensus.cc:2454] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69b854c27145c19fe37aaacfda117e in term 1.
I20260928 22:53:44.506788 16422 raft_consensus.cc:2454] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69b854c27145c19fe37aaacfda117e in term 1.
I20260928 22:53:44.507001 16493 leader_election.cc:302] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69b854c27145c19fe37aaacfda117e, 1535b5448d064ccda6f5ff6f99e10d9f; no voters:
I20260928 22:53:44.507117 16639 raft_consensus.cc:2781] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:44.507297 16639 raft_consensus.cc:686] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [term 1 LEADER]: Becoming Leader. State: Replica: 0a69b854c27145c19fe37aaacfda117e, State: Running, Role: LEADER
I20260928 22:53:44.507396 16639 consensus_queue.cc:237] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.508301 16124 catalog_manager.cc:5495] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a69b854c27145c19fe37aaacfda117e (127.14.14.131). New cstate: current_term: 1 leader_uuid: "0a69b854c27145c19fe37aaacfda117e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } health_report { overall_health: HEALTHY } } }
Sep 28 22:53:44 dist-test-slave-kzfr krb5kdc[16080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636023, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:44.580979 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.574292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.14.14.190:38387)
0928 22:53:44.574344 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:44.574347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:44.574465 (+ 118us) server_negotiation.cc:407] Connection header received
0928 22:53:44.574535 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:44.574536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:44.574549 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:44.574616 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:44.575111 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.575595 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.576289 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.576470 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.576553 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.578624 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:44.578625 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:44.578625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:44.578632 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:44.579798 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:44.580174 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:44.580175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:44.580176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:44.580229 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:44.580425 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:44.580426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:44.580426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:44.580542 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:44.580616 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:44.580771 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 22:53:44.580871 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:53:44.581049 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.574227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38387 (local address 127.0.0.1:58970)
0928 22:53:44.574379 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:44.574390 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:44.574481 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:44.574618 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:44.574619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:44.574967 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:44.575021 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.575025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.575677 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.575678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.576207 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.576208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.576553 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.576554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.576608 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.577209 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:44.577213 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:44.578535 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:44.579890 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:44.579897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:44.579899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:44.580085 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:44.580294 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:44.580295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:44.580295 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:44.580352 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:44.580607 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:44.580614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:44.580664 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 22:53:44.580749 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 22:53:44.580928 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":83,"thread_start_us":75,"threads_started":1}
I20260928 22:53:44.583130 16124 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:58970:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 22:53:44.583349 16124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:53:44.584837 16402 tablet_service.cc:1467] Processing CreateTablet for tablet 9aeacbfb7c844f0496453ec6140b93fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4fdc0923462f47239ae885bfed5338b9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:53:44.584888 16262 tablet_service.cc:1467] Processing CreateTablet for tablet 9aeacbfb7c844f0496453ec6140b93fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4fdc0923462f47239ae885bfed5338b9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:53:44.584919 16542 tablet_service.cc:1467] Processing CreateTablet for tablet 9aeacbfb7c844f0496453ec6140b93fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4fdc0923462f47239ae885bfed5338b9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:53:44.585029 16402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aeacbfb7c844f0496453ec6140b93fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:44.585062 16262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aeacbfb7c844f0496453ec6140b93fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:44.585062 16542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aeacbfb7c844f0496453ec6140b93fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:44.586323 16633 tablet_bootstrap.cc:492] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f: Bootstrap starting.
I20260928 22:53:44.586580 16632 tablet_bootstrap.cc:492] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e: Bootstrap starting.
I20260928 22:53:44.586589 16631 tablet_bootstrap.cc:492] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c: Bootstrap starting.
I20260928 22:53:44.586751 16633 tablet_bootstrap.cc:654] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:44.587002 16631 tablet_bootstrap.cc:654] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:44.587003 16632 tablet_bootstrap.cc:654] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:44.587652 16631 tablet_bootstrap.cc:492] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c: No bootstrap required, opened a new log
I20260928 22:53:44.587713 16632 tablet_bootstrap.cc:492] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e: No bootstrap required, opened a new log
I20260928 22:53:44.587710 16631 ts_tablet_manager.cc:1397] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:44.587752 16632 ts_tablet_manager.cc:1397] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:44.587899 16633 tablet_bootstrap.cc:492] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f: No bootstrap required, opened a new log
I20260928 22:53:44.587877 16631 raft_consensus.cc:348] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.587896 16632 raft_consensus.cc:348] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.587955 16631 raft_consensus.cc:374] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:44.587962 16632 raft_consensus.cc:374] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:44.587967 16633 ts_tablet_manager.cc:1397] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:53:44.587981 16631 raft_consensus.cc:729] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4fff65288a54c3fb142b3c7fdea7d7c, State: Initialized, Role: FOLLOWER
I20260928 22:53:44.587985 16632 raft_consensus.cc:729] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69b854c27145c19fe37aaacfda117e, State: Initialized, Role: FOLLOWER
I20260928 22:53:44.588028 16632 consensus_queue.cc:260] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.588038 16631 consensus_queue.cc:260] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.588115 16632 ts_tablet_manager.cc:1428] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:44.588135 16631 ts_tablet_manager.cc:1428] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:44.588116 16633 raft_consensus.cc:348] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.588213 16633 raft_consensus.cc:374] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:44.588241 16633 raft_consensus.cc:729] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1535b5448d064ccda6f5ff6f99e10d9f, State: Initialized, Role: FOLLOWER
I20260928 22:53:44.588235 16639 tablet_replica.cc:439] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } } }
I20260928 22:53:44.588305 16639 tablet_replica.cc:442] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:53:44.588297 16633 consensus_queue.cc:260] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.588387 16633 ts_tablet_manager.cc:1428] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:44.588434 16638 tablet_replica.cc:439] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } } }
I20260928 22:53:44.588517 16638 tablet_replica.cc:442] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:53:44.588712 16637 tablet_replica.cc:439] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } } }
I20260928 22:53:44.588820 16637 tablet_replica.cc:442] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:53:44.605291 16639 raft_consensus.cc:482] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:53:44.605381 16639 raft_consensus.cc:504] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.605571 16639 leader_election.cc:288] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4fff65288a54c3fb142b3c7fdea7d7c (127.14.14.129:36883), 1535b5448d064ccda6f5ff6f99e10d9f (127.14.14.130:45111)
I20260928 22:53:44.605789 16422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeacbfb7c844f0496453ec6140b93fa" candidate_uuid: "0a69b854c27145c19fe37aaacfda117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" is_pre_election: true
I20260928 22:53:44.605798 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeacbfb7c844f0496453ec6140b93fa" candidate_uuid: "0a69b854c27145c19fe37aaacfda117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" is_pre_election: true
I20260928 22:53:44.605868 16265 raft_consensus.cc:2454] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a69b854c27145c19fe37aaacfda117e in term 0.
I20260928 22:53:44.605890 16422 raft_consensus.cc:2454] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a69b854c27145c19fe37aaacfda117e in term 0.
I20260928 22:53:44.606024 16494 leader_election.cc:302] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69b854c27145c19fe37aaacfda117e, d4fff65288a54c3fb142b3c7fdea7d7c; no voters:
I20260928 22:53:44.606223 16639 raft_consensus.cc:2781] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:53:44.606271 16639 raft_consensus.cc:482] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:53:44.606287 16639 raft_consensus.cc:3037] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:44.607023 16639 raft_consensus.cc:504] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.607165 16639 leader_election.cc:288] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [CANDIDATE]: Term 1 election: Requested vote from peers d4fff65288a54c3fb142b3c7fdea7d7c (127.14.14.129:36883), 1535b5448d064ccda6f5ff6f99e10d9f (127.14.14.130:45111)
I20260928 22:53:44.607309 16422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeacbfb7c844f0496453ec6140b93fa" candidate_uuid: "0a69b854c27145c19fe37aaacfda117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1535b5448d064ccda6f5ff6f99e10d9f"
I20260928 22:53:44.607313 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeacbfb7c844f0496453ec6140b93fa" candidate_uuid: "0a69b854c27145c19fe37aaacfda117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c"
I20260928 22:53:44.607376 16265 raft_consensus.cc:3037] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:44.607388 16422 raft_consensus.cc:3037] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:44.608019 16265 raft_consensus.cc:2454] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69b854c27145c19fe37aaacfda117e in term 1.
I20260928 22:53:44.608160 16422 raft_consensus.cc:2454] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69b854c27145c19fe37aaacfda117e in term 1.
I20260928 22:53:44.608173 16494 leader_election.cc:302] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69b854c27145c19fe37aaacfda117e, d4fff65288a54c3fb142b3c7fdea7d7c; no voters:
I20260928 22:53:44.608301 16639 raft_consensus.cc:2781] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:44.608356 16639 raft_consensus.cc:686] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [term 1 LEADER]: Becoming Leader. State: Replica: 0a69b854c27145c19fe37aaacfda117e, State: Running, Role: LEADER
I20260928 22:53:44.608412 16639 consensus_queue.cc:237] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.608731 16641 tablet_replica.cc:439] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e: TxnStatusTablet state changed. Reason: New leader 0a69b854c27145c19fe37aaacfda117e. Latest consensus state: current_term: 1 leader_uuid: "0a69b854c27145c19fe37aaacfda117e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } } }
I20260928 22:53:44.608812 16641 tablet_replica.cc:442] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:53:44.609009 16653 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 22:53:44.609006 16123 catalog_manager.cc:5495] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a69b854c27145c19fe37aaacfda117e (127.14.14.131). New cstate: current_term: 1 leader_uuid: "0a69b854c27145c19fe37aaacfda117e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } health_report { overall_health: HEALTHY } } }
I20260928 22:53:44.609154 16653 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 22:53:44.617574 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.615177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.14.14.131:40717)
0928 22:53:44.615238 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:44.615241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:44.615274 (+ 33us) server_negotiation.cc:407] Connection header received
0928 22:53:44.615370 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:44.615370 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:44.615387 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:44.615445 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:44.615554 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.616038 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.616780 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.616985 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.617068 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.617251 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:44.617349 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:44.617435 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:44.617487 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 22:53:44.617507 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:53:44.617619 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.615117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:40717 (local address 127.0.0.1:55980)
0928 22:53:44.615230 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:44.615242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:44.615294 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:44.615454 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:44.615455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:44.615457 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:53:44.615503 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.615506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.616121 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.616122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.616699 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.616700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.617069 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.617070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.617135 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.617163 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:53:44.617414 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:53:44.617424 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 22:53:44.617481 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:53:44.617525 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 22:53:44.619225 16265 raft_consensus.cc:1261] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Refusing update from remote peer 0a69b854c27145c19fe37aaacfda117e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:44.619220 16422 raft_consensus.cc:1261] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Refusing update from remote peer 0a69b854c27145c19fe37aaacfda117e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:44.619503 16639 consensus_queue.cc:1035] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:44.619604 16641 consensus_queue.cc:1035] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:44.620693 16637 tablet_replica.cc:439] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f: TxnStatusTablet state changed. Reason: New leader 0a69b854c27145c19fe37aaacfda117e. Latest consensus state: current_term: 1 leader_uuid: "0a69b854c27145c19fe37aaacfda117e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } } }
I20260928 22:53:44.620775 16637 tablet_replica.cc:442] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:53:44.620759 16638 tablet_replica.cc:439] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c: TxnStatusTablet state changed. Reason: New leader 0a69b854c27145c19fe37aaacfda117e. Latest consensus state: current_term: 1 leader_uuid: "0a69b854c27145c19fe37aaacfda117e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } } }
I20260928 22:53:44.620842 16638 tablet_replica.cc:442] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:53:44.621835 16639 tablet_replica.cc:439] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a69b854c27145c19fe37aaacfda117e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } } }
I20260928 22:53:44.621918 16639 tablet_replica.cc:442] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:53:44.621974 16656 mvcc.cc:205] Tried to move back new op lower bound from 7334445156837761024 to 7334445156796358656. Current Snapshot: MvccSnapshot[applied={T|T < 7334445156837761024}]
I20260928 22:53:44.622042 16638 tablet_replica.cc:439] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a69b854c27145c19fe37aaacfda117e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } } }
I20260928 22:53:44.622123 16638 tablet_replica.cc:442] T 9aeacbfb7c844f0496453ec6140b93fa P d4fff65288a54c3fb142b3c7fdea7d7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:53:44.621850 16641 tablet_replica.cc:439] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a69b854c27145c19fe37aaacfda117e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } } }
I20260928 22:53:44.622555 16641 tablet_replica.cc:442] T 9aeacbfb7c844f0496453ec6140b93fa P 0a69b854c27145c19fe37aaacfda117e: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:53:44.622593 16637 tablet_replica.cc:439] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a69b854c27145c19fe37aaacfda117e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } } }
I20260928 22:53:44.622689 16637 tablet_replica.cc:442] T 9aeacbfb7c844f0496453ec6140b93fa P 1535b5448d064ccda6f5ff6f99e10d9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:53:44.626977 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.624793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.14.14.131:40717)
0928 22:53:44.624830 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:44.624832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:44.624840 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:53:44.624859 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:44.624860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:44.624869 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:44.624921 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:44.625038 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.625475 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.626299 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.626501 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.626585 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.626736 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:44.626765 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:44.626831 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:44.626891 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 22:53:44.626907 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:53:44.632207 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.629765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:48929 (local address 127.14.14.131:40717)
0928 22:53:44.629804 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:44.629806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:44.629845 (+ 39us) server_negotiation.cc:407] Connection header received
0928 22:53:44.629889 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:44.629890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:44.629899 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:44.629962 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:44.630117 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.630816 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.631546 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.631746 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.631815 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.631989 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:44.632017 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:44.632058 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:44.632111 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 22:53:44.632130 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:53:44.632207 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:44.629722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:40717 (local address 127.14.14.131:48929)
0928 22:53:44.629807 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:44.629816 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:44.629852 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:44.629996 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:44.629997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:44.629998 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:53:44.630051 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.630052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.630914 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.630915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.631461 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:44.631463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:44.631825 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:44.631826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:44.631883 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:44.631897 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:53:44.632069 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:53:44.632070 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 22:53:44.632111 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 22:53:44.632147 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 22:53:44.634541 16422 raft_consensus.cc:1261] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Refusing update from remote peer 0a69b854c27145c19fe37aaacfda117e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:44.634562 16265 raft_consensus.cc:1261] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Refusing update from remote peer 0a69b854c27145c19fe37aaacfda117e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:44.634730 16641 consensus_queue.cc:1035] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:44.634824 16661 consensus_queue.cc:1035] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:44.636402 16657 mvcc.cc:205] Tried to move back new op lower bound from 7334445156901756928 to 7334445156382920704. Current Snapshot: MvccSnapshot[applied={T|T < 7334445156901756928}]
I20260928 22:53:44.648890 16124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:44.648972 16124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:44.649703 16124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc: Sending DeleteTablet for 3 replicas of tablet 3a9a3948859a48e78a20f8d9d8e7c714
I20260928 22:53:44.649966 16262 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a9a3948859a48e78a20f8d9d8e7c714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:44 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:40070
I20260928 22:53:44.649996 16402 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a9a3948859a48e78a20f8d9d8e7c714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:44 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:37766
I20260928 22:53:44.650180 16541 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a9a3948859a48e78a20f8d9d8e7c714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:44 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:55972
I20260928 22:53:44.650213 16671 tablet_replica.cc:330] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c: stopping tablet replica
I20260928 22:53:44.650241 16672 tablet_replica.cc:330] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f: stopping tablet replica
I20260928 22:53:44.650312 16672 raft_consensus.cc:2229] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:44.650313 16671 raft_consensus.cc:2229] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:44.650362 16672 raft_consensus.cc:2258] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:44.650362 16671 raft_consensus.cc:2258] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:44.650731 16671 ts_tablet_manager.cc:1905] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:44.650822 16124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:53:44.650959 16124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:53:44.651281 16672 ts_tablet_manager.cc:1905] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:44.652143 16402 tablet_service.cc:1467] Processing CreateTablet for tablet 7bb3b77cb95149f597970bb8e95b22b4 (DEFAULT_TABLE table=test-table [id=b1721de34b024628b80f35e0b8c75631]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:44.652462 16262 tablet_service.cc:1467] Processing CreateTablet for tablet 7bb3b77cb95149f597970bb8e95b22b4 (DEFAULT_TABLE table=test-table [id=b1721de34b024628b80f35e0b8c75631]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:44.652479 16673 tablet_replica.cc:330] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e: stopping tablet replica
I20260928 22:53:44.652566 16262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb3b77cb95149f597970bb8e95b22b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:44.652675 16673 raft_consensus.cc:2229] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:53:44.652814 16673 raft_consensus.cc:2258] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:44.653187 16671 ts_tablet_manager.cc:1918] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:53:44.653265 16671 log.cc:1196] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/wal/wals/3a9a3948859a48e78a20f8d9d8e7c714
I20260928 22:53:44.653414 16673 ts_tablet_manager.cc:1905] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:44.653528 16671 ts_tablet_manager.cc:1939] T 3a9a3948859a48e78a20f8d9d8e7c714 P d4fff65288a54c3fb142b3c7fdea7d7c: Deleting consensus metadata
I20260928 22:53:44.653925 16541 tablet_service.cc:1467] Processing CreateTablet for tablet 7bb3b77cb95149f597970bb8e95b22b4 (DEFAULT_TABLE table=test-table [id=b1721de34b024628b80f35e0b8c75631]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:44.654059 16541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb3b77cb95149f597970bb8e95b22b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:44.655086 16632 tablet_bootstrap.cc:492] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e: Bootstrap starting.
I20260928 22:53:44.655128 16631 tablet_bootstrap.cc:492] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c: Bootstrap starting.
I20260928 22:53:44.655182 16673 ts_tablet_manager.cc:1918] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:53:44.655243 16673 log.cc:1196] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/3a9a3948859a48e78a20f8d9d8e7c714
I20260928 22:53:44.655494 16631 tablet_bootstrap.cc:654] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:44.655494 16632 tablet_bootstrap.cc:654] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:44.655508 16673 ts_tablet_manager.cc:1939] T 3a9a3948859a48e78a20f8d9d8e7c714 P 0a69b854c27145c19fe37aaacfda117e: Deleting consensus metadata
I20260928 22:53:44.655999 16402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb3b77cb95149f597970bb8e95b22b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:44.656311 16632 tablet_bootstrap.cc:492] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e: No bootstrap required, opened a new log
I20260928 22:53:44.656351 16632 ts_tablet_manager.cc:1397] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:44.656484 16632 raft_consensus.cc:348] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.656553 16632 raft_consensus.cc:374] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:44.656572 16632 raft_consensus.cc:729] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69b854c27145c19fe37aaacfda117e, State: Initialized, Role: FOLLOWER
I20260928 22:53:44.656611 16632 consensus_queue.cc:260] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.656721 16632 ts_tablet_manager.cc:1428] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:44.656714 16672 ts_tablet_manager.cc:1918] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:53:44.657032 16672 log.cc:1196] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/3a9a3948859a48e78a20f8d9d8e7c714
I20260928 22:53:44.657346 16672 ts_tablet_manager.cc:1939] T 3a9a3948859a48e78a20f8d9d8e7c714 P 1535b5448d064ccda6f5ff6f99e10d9f: Deleting consensus metadata
I20260928 22:53:44.657354 16633 tablet_bootstrap.cc:492] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f: Bootstrap starting.
I20260928 22:53:44.657541 16110 catalog_manager.cc:4841] TS d4fff65288a54c3fb142b3c7fdea7d7c (127.14.14.129:36883): tablet 3a9a3948859a48e78a20f8d9d8e7c714 (table test-table [id=f652a5032deb48ba82d164a1b6758b7a]) successfully deleted
I20260928 22:53:44.657768 16109 catalog_manager.cc:4841] TS 1535b5448d064ccda6f5ff6f99e10d9f (127.14.14.130:45111): tablet 3a9a3948859a48e78a20f8d9d8e7c714 (table test-table [id=f652a5032deb48ba82d164a1b6758b7a]) successfully deleted
I20260928 22:53:44.657811 16633 tablet_bootstrap.cc:654] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:44.657829 16110 catalog_manager.cc:4841] TS 0a69b854c27145c19fe37aaacfda117e (127.14.14.131:40717): tablet 3a9a3948859a48e78a20f8d9d8e7c714 (table test-table [id=f652a5032deb48ba82d164a1b6758b7a]) successfully deleted
I20260928 22:53:44.658365 16631 tablet_bootstrap.cc:492] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c: No bootstrap required, opened a new log
I20260928 22:53:44.658420 16631 ts_tablet_manager.cc:1397] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 22:53:44.658586 16633 tablet_bootstrap.cc:492] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f: No bootstrap required, opened a new log
I20260928 22:53:44.658571 16631 raft_consensus.cc:348] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.658620 16633 ts_tablet_manager.cc:1397] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:44.658639 16631 raft_consensus.cc:374] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:44.658660 16631 raft_consensus.cc:729] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4fff65288a54c3fb142b3c7fdea7d7c, State: Initialized, Role: FOLLOWER
I20260928 22:53:44.658715 16631 consensus_queue.cc:260] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.658761 16633 raft_consensus.cc:348] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.658799 16631 ts_tablet_manager.cc:1428] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:44.658823 16633 raft_consensus.cc:374] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:44.658845 16633 raft_consensus.cc:729] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1535b5448d064ccda6f5ff6f99e10d9f, State: Initialized, Role: FOLLOWER
I20260928 22:53:44.658885 16633 consensus_queue.cc:260] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.659010 16633 ts_tablet_manager.cc:1428] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:44.673758 16639 raft_consensus.cc:482] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:53:44.673861 16639 raft_consensus.cc:504] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.674057 16639 leader_election.cc:288] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4fff65288a54c3fb142b3c7fdea7d7c (127.14.14.129:36883), 1535b5448d064ccda6f5ff6f99e10d9f (127.14.14.130:45111)
I20260928 22:53:44.674254 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb3b77cb95149f597970bb8e95b22b4" candidate_uuid: "0a69b854c27145c19fe37aaacfda117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" is_pre_election: true
I20260928 22:53:44.674254 16422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb3b77cb95149f597970bb8e95b22b4" candidate_uuid: "0a69b854c27145c19fe37aaacfda117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" is_pre_election: true
I20260928 22:53:44.674345 16265 raft_consensus.cc:2454] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a69b854c27145c19fe37aaacfda117e in term 0.
I20260928 22:53:44.674345 16422 raft_consensus.cc:2454] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a69b854c27145c19fe37aaacfda117e in term 0.
I20260928 22:53:44.674485 16494 leader_election.cc:302] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69b854c27145c19fe37aaacfda117e, d4fff65288a54c3fb142b3c7fdea7d7c; no voters:
I20260928 22:53:44.674579 16639 raft_consensus.cc:2781] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:53:44.674618 16639 raft_consensus.cc:482] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:53:44.674641 16639 raft_consensus.cc:3037] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:44.675271 16639 raft_consensus.cc:504] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.675402 16639 leader_election.cc:288] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [CANDIDATE]: Term 1 election: Requested vote from peers d4fff65288a54c3fb142b3c7fdea7d7c (127.14.14.129:36883), 1535b5448d064ccda6f5ff6f99e10d9f (127.14.14.130:45111)
I20260928 22:53:44.675534 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb3b77cb95149f597970bb8e95b22b4" candidate_uuid: "0a69b854c27145c19fe37aaacfda117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c"
I20260928 22:53:44.675602 16265 raft_consensus.cc:3037] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:44.675570 16422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb3b77cb95149f597970bb8e95b22b4" candidate_uuid: "0a69b854c27145c19fe37aaacfda117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1535b5448d064ccda6f5ff6f99e10d9f"
I20260928 22:53:44.675657 16422 raft_consensus.cc:3037] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:44.676179 16265 raft_consensus.cc:2454] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69b854c27145c19fe37aaacfda117e in term 1.
I20260928 22:53:44.676334 16494 leader_election.cc:302] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69b854c27145c19fe37aaacfda117e, d4fff65288a54c3fb142b3c7fdea7d7c; no voters:
I20260928 22:53:44.676411 16639 raft_consensus.cc:2781] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:44.676409 16422 raft_consensus.cc:2454] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69b854c27145c19fe37aaacfda117e in term 1.
I20260928 22:53:44.676455 16639 raft_consensus.cc:686] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [term 1 LEADER]: Becoming Leader. State: Replica: 0a69b854c27145c19fe37aaacfda117e, State: Running, Role: LEADER
I20260928 22:53:44.676502 16639 consensus_queue.cc:237] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:44.677057 16123 catalog_manager.cc:5495] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a69b854c27145c19fe37aaacfda117e (127.14.14.131). New cstate: current_term: 1 leader_uuid: "0a69b854c27145c19fe37aaacfda117e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } health_report { overall_health: HEALTHY } } }
I20260928 22:53:44.684712 16265 raft_consensus.cc:1261] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Refusing update from remote peer 0a69b854c27145c19fe37aaacfda117e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:44.684762 16421 raft_consensus.cc:1261] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Refusing update from remote peer 0a69b854c27145c19fe37aaacfda117e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:44.684947 16641 consensus_queue.cc:1035] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:44.685041 16661 consensus_queue.cc:1035] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:44.686470 16658 mvcc.cc:205] Tried to move back new op lower bound from 7334445157107286016 to 7334445157075165184. Current Snapshot: MvccSnapshot[applied={T|T < 7334445157107286016}]
W20260928 22:53:44.689512 16656 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 22:53:46.209759 16123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:46.209874 16123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:46.210855 16123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc: Sending DeleteTablet for 3 replicas of tablet 7bb3b77cb95149f597970bb8e95b22b4
I20260928 22:53:46.211185 16262 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bb3b77cb95149f597970bb8e95b22b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:46 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:40070
I20260928 22:53:46.211184 16402 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bb3b77cb95149f597970bb8e95b22b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:46 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:37766
I20260928 22:53:46.211390 16699 tablet_replica.cc:330] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f: stopping tablet replica
I20260928 22:53:46.211464 16541 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bb3b77cb95149f597970bb8e95b22b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:46 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:55972
I20260928 22:53:46.211499 16699 raft_consensus.cc:2229] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:46.211541 16699 raft_consensus.cc:2258] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:46.211804 16699 ts_tablet_manager.cc:1905] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:46.213824 16699 ts_tablet_manager.cc:1918] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 22:53:46.213896 16699 log.cc:1196] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/7bb3b77cb95149f597970bb8e95b22b4
I20260928 22:53:46.214119 16699 ts_tablet_manager.cc:1939] T 7bb3b77cb95149f597970bb8e95b22b4 P 1535b5448d064ccda6f5ff6f99e10d9f: Deleting consensus metadata
I20260928 22:53:46.214713 16109 catalog_manager.cc:4841] TS 1535b5448d064ccda6f5ff6f99e10d9f (127.14.14.130:45111): tablet 7bb3b77cb95149f597970bb8e95b22b4 (table test-table [id=b1721de34b024628b80f35e0b8c75631]) successfully deleted
I20260928 22:53:46.216585 16700 tablet_replica.cc:330] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c: stopping tablet replica
I20260928 22:53:46.216663 16700 raft_consensus.cc:2229] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:46.216709 16700 raft_consensus.cc:2258] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:46.216836 16701 tablet_replica.cc:330] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e: stopping tablet replica
I20260928 22:53:46.216899 16701 raft_consensus.cc:2229] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:53:46.216971 16701 raft_consensus.cc:2258] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:46.216979 16700 ts_tablet_manager.cc:1905] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:46.217833 16701 ts_tablet_manager.cc:1905] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:46.218583 16700 ts_tablet_manager.cc:1918] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 22:53:46.218659 16700 log.cc:1196] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/wal/wals/7bb3b77cb95149f597970bb8e95b22b4
I20260928 22:53:46.218881 16700 ts_tablet_manager.cc:1939] T 7bb3b77cb95149f597970bb8e95b22b4 P d4fff65288a54c3fb142b3c7fdea7d7c: Deleting consensus metadata
I20260928 22:53:46.219223 16110 catalog_manager.cc:4841] TS d4fff65288a54c3fb142b3c7fdea7d7c (127.14.14.129:36883): tablet 7bb3b77cb95149f597970bb8e95b22b4 (table test-table [id=b1721de34b024628b80f35e0b8c75631]) successfully deleted
I20260928 22:53:46.219293 16701 ts_tablet_manager.cc:1918] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 22:53:46.219348 16701 log.cc:1196] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/7bb3b77cb95149f597970bb8e95b22b4
I20260928 22:53:46.219552 16701 ts_tablet_manager.cc:1939] T 7bb3b77cb95149f597970bb8e95b22b4 P 0a69b854c27145c19fe37aaacfda117e: Deleting consensus metadata
I20260928 22:53:46.220057 16110 catalog_manager.cc:4841] TS 0a69b854c27145c19fe37aaacfda117e (127.14.14.131:40717): tablet 7bb3b77cb95149f597970bb8e95b22b4 (table test-table [id=b1721de34b024628b80f35e0b8c75631]) successfully deleted
Sep 28 22:53:46 dist-test-slave-kzfr krb5kdc[16080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636026, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 22:53:46 dist-test-slave-kzfr krb5kdc[16080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636026, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:46.237676 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:46.230036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.14.14.190:38387)
0928 22:53:46.230176 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:46.230178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:46.230256 (+ 78us) server_negotiation.cc:407] Connection header received
0928 22:53:46.230298 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:46.230299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:46.230310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:46.230369 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:46.230907 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.231411 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.232263 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.232454 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.232505 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:46.234811 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:46.234813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:46.234813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:46.234822 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:46.236286 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:46.236644 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:46.236645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:46.236645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:46.236700 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:46.236863 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:46.236863 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:46.236864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:46.236960 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:46.237033 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:46.237232 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 22:53:46.237434 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260928 22:53:46.239094 16123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:53:46.239276 16123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:53:46.240643 16541 tablet_service.cc:1467] Processing CreateTablet for tablet fb9b3077bdd540eba971594d19c4b38b (DEFAULT_TABLE table=test-table [id=bde7f5ac8b9d41628cc01f87b319fd43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:46.240725 16262 tablet_service.cc:1467] Processing CreateTablet for tablet fb9b3077bdd540eba971594d19c4b38b (DEFAULT_TABLE table=test-table [id=bde7f5ac8b9d41628cc01f87b319fd43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:46.240775 16541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9b3077bdd540eba971594d19c4b38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:46.240747 16402 tablet_service.cc:1467] Processing CreateTablet for tablet fb9b3077bdd540eba971594d19c4b38b (DEFAULT_TABLE table=test-table [id=bde7f5ac8b9d41628cc01f87b319fd43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:46.240854 16262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9b3077bdd540eba971594d19c4b38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:46.240859 16402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9b3077bdd540eba971594d19c4b38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:46.241977 16710 tablet_bootstrap.cc:492] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c: Bootstrap starting.
I20260928 22:53:46.242457 16711 tablet_bootstrap.cc:492] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f: Bootstrap starting.
I20260928 22:53:46.242550 16710 tablet_bootstrap.cc:654] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:46.242798 16711 tablet_bootstrap.cc:654] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:46.242893 16712 tablet_bootstrap.cc:492] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e: Bootstrap starting.
I20260928 22:53:46.243251 16712 tablet_bootstrap.cc:654] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:46.243310 16710 tablet_bootstrap.cc:492] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c: No bootstrap required, opened a new log
I20260928 22:53:46.243355 16710 ts_tablet_manager.cc:1397] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:46.243535 16710 raft_consensus.cc:348] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.243619 16710 raft_consensus.cc:374] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:46.243635 16710 raft_consensus.cc:729] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4fff65288a54c3fb142b3c7fdea7d7c, State: Initialized, Role: FOLLOWER
I20260928 22:53:46.243701 16710 consensus_queue.cc:260] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.243878 16710 ts_tablet_manager.cc:1428] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:46.243944 16712 tablet_bootstrap.cc:492] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e: No bootstrap required, opened a new log
I20260928 22:53:46.243996 16712 ts_tablet_manager.cc:1397] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:46.244114 16712 raft_consensus.cc:348] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.244199 16712 raft_consensus.cc:374] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:46.244226 16712 raft_consensus.cc:729] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69b854c27145c19fe37aaacfda117e, State: Initialized, Role: FOLLOWER
I20260928 22:53:46.244298 16712 consensus_queue.cc:260] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.244509 16711 tablet_bootstrap.cc:492] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f: No bootstrap required, opened a new log
I20260928 22:53:46.244562 16711 ts_tablet_manager.cc:1397] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:53:46.244575 16712 ts_tablet_manager.cc:1428] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:46.244719 16711 raft_consensus.cc:348] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.244798 16711 raft_consensus.cc:374] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:46.244823 16711 raft_consensus.cc:729] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1535b5448d064ccda6f5ff6f99e10d9f, State: Initialized, Role: FOLLOWER
I20260928 22:53:46.244885 16711 consensus_queue.cc:260] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.245066 16711 ts_tablet_manager.cc:1428] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:46.465847 16717 raft_consensus.cc:482] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:53:46.465972 16717 raft_consensus.cc:504] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.466341 16717 leader_election.cc:288] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4fff65288a54c3fb142b3c7fdea7d7c (127.14.14.129:36883), 0a69b854c27145c19fe37aaacfda117e (127.14.14.131:40717)
I20260928 22:53:46.469552 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:46.466631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:37265 (local address 127.14.14.129:36883)
0928 22:53:46.466807 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:46.466810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:46.466955 (+ 145us) server_negotiation.cc:407] Connection header received
0928 22:53:46.466992 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:46.466993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:46.467005 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:46.467066 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:46.467281 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.467787 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.468801 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.469178 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.469274 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:46.469284 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:46.469436 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 22:53:46.469478 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260928 22:53:46.469551 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:46.466551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:36883 (local address 127.14.14.130:37265)
0928 22:53:46.466797 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:46.466809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:46.466894 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:46.467119 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:46.467119 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:46.467122 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:46.467209 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:46.467211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.467884 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.467885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:46.468702 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:46.468704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.469273 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.469274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:46.469352 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:46.469355 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:53:46.469418 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 22:53:46.469459 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":166,"threads_started":1}
I20260928 22:53:46.469854 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:46.466843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:40717 (local address 127.14.14.130:51109)
0928 22:53:46.466967 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:46.466977 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:46.467032 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:46.467183 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:46.467184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:46.467185 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:46.467537 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:46.467538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.468271 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.468272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:46.468984 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:46.468986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.469532 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.469533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:46.469656 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:46.469660 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:46.469726 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 22:53:46.469776 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260928 22:53:46.469820 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9b3077bdd540eba971594d19c4b38b" candidate_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" is_pre_election: true
I20260928 22:53:46.469897 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:46.466909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:51109 (local address 127.14.14.131:40717)
0928 22:53:46.467044 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:46.467046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:46.467055 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:53:46.467085 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:46.467085 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:46.467097 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:46.467169 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:46.467623 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.468188 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.469074 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.469441 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.469522 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:46.469530 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:46.469789 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 22:53:46.469831 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260928 22:53:46.469918 16265 raft_consensus.cc:2454] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1535b5448d064ccda6f5ff6f99e10d9f in term 0.
I20260928 22:53:46.470177 16354 leader_election.cc:302] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1535b5448d064ccda6f5ff6f99e10d9f, d4fff65288a54c3fb142b3c7fdea7d7c; no voters:
I20260928 22:53:46.470209 16562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9b3077bdd540eba971594d19c4b38b" candidate_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69b854c27145c19fe37aaacfda117e" is_pre_election: true
I20260928 22:53:46.470327 16717 raft_consensus.cc:2781] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:53:46.470368 16562 raft_consensus.cc:2454] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1535b5448d064ccda6f5ff6f99e10d9f in term 0.
I20260928 22:53:46.470382 16717 raft_consensus.cc:482] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:53:46.470410 16717 raft_consensus.cc:3037] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:46.471091 16717 raft_consensus.cc:504] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.471240 16717 leader_election.cc:288] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [CANDIDATE]: Term 1 election: Requested vote from peers d4fff65288a54c3fb142b3c7fdea7d7c (127.14.14.129:36883), 0a69b854c27145c19fe37aaacfda117e (127.14.14.131:40717)
I20260928 22:53:46.471390 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9b3077bdd540eba971594d19c4b38b" candidate_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c"
I20260928 22:53:46.471458 16562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9b3077bdd540eba971594d19c4b38b" candidate_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69b854c27145c19fe37aaacfda117e"
I20260928 22:53:46.471494 16265 raft_consensus.cc:3037] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:46.471524 16562 raft_consensus.cc:3037] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:46.472245 16265 raft_consensus.cc:2454] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1535b5448d064ccda6f5ff6f99e10d9f in term 1.
I20260928 22:53:46.472254 16562 raft_consensus.cc:2454] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1535b5448d064ccda6f5ff6f99e10d9f in term 1.
I20260928 22:53:46.472415 16354 leader_election.cc:302] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69b854c27145c19fe37aaacfda117e, 1535b5448d064ccda6f5ff6f99e10d9f; no voters:
I20260928 22:53:46.472522 16717 raft_consensus.cc:2781] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:46.472668 16717 raft_consensus.cc:686] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 LEADER]: Becoming Leader. State: Replica: 1535b5448d064ccda6f5ff6f99e10d9f, State: Running, Role: LEADER
I20260928 22:53:46.472734 16717 consensus_queue.cc:237] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.473528 16124 catalog_manager.cc:5495] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1535b5448d064ccda6f5ff6f99e10d9f (127.14.14.130). New cstate: current_term: 1 leader_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } health_report { overall_health: UNKNOWN } } }
I20260928 22:53:46.529711 16725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:46.526849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.14.14.130:45111)
0928 22:53:46.527020 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:46.527022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:46.527036 (+ 14us) server_negotiation.cc:407] Connection header received
0928 22:53:46.527072 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:46.527073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:46.527091 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:46.527182 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:46.527347 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.527966 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.528759 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.529030 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.529124 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:46.529314 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:46.529432 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:46.529521 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:46.529562 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 22:53:46.529605 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260928 22:53:46.534329 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:46.532204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:55629 (local address 127.14.14.131:40717)
0928 22:53:46.532258 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:46.532260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:46.532361 (+ 101us) server_negotiation.cc:407] Connection header received
0928 22:53:46.532409 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:46.532409 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:46.532417 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:46.532463 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:46.532571 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.533045 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.533801 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.533976 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.534025 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:46.534155 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:46.534177 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:46.534210 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:46.534262 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 22:53:46.534276 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 22:53:46.534371 16728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:46.532089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:40717 (local address 127.14.14.130:55629)
0928 22:53:46.532281 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:46.532293 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:46.532355 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:46.532479 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:46.532480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:46.532481 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:53:46.532534 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:46.532535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.533103 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.533103 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:46.533729 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:46.533730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.534037 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.534038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:46.534083 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:46.534096 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:53:46.534225 (+ 129us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:53:46.534226 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 22:53:46.534259 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 22:53:46.534292 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260928 22:53:46.536401 16265 raft_consensus.cc:1261] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Refusing update from remote peer 1535b5448d064ccda6f5ff6f99e10d9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:46.536417 16562 raft_consensus.cc:1261] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Refusing update from remote peer 1535b5448d064ccda6f5ff6f99e10d9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:46.536716 16722 consensus_queue.cc:1035] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:46.536824 16717 consensus_queue.cc:1035] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:46.544375 16725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:46.541972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:50521 (local address 127.14.14.130:45111)
0928 22:53:46.542009 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:46.542011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:46.542139 (+ 128us) server_negotiation.cc:407] Connection header received
0928 22:53:46.542235 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:46.542236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:46.542245 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:46.542275 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:46.542435 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.542896 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.543649 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.543878 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.543948 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:46.544124 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:46.544157 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:46.544220 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:46.544272 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 22:53:46.544295 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:53:46.544411 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:46.541953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:45111 (local address 127.14.14.131:50521)
0928 22:53:46.542081 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:46.542093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:46.542152 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:46.542335 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:46.542336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:46.542338 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:53:46.542383 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:46.542385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.542980 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.542981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:46.543563 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:46.543564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:46.543973 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:46.543974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:46.544052 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:46.544057 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:53:46.544224 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:53:46.544226 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:53:46.544269 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 22:53:46.544325 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":82,"threads_started":1}
I20260928 22:53:46.551110 16123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:46.551210 16123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:46.551931 16123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc: Sending DeleteTablet for 3 replicas of tablet fb9b3077bdd540eba971594d19c4b38b
I20260928 22:53:46.552172 16262 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9b3077bdd540eba971594d19c4b38b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:46 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:40070
I20260928 22:53:46.552196 16541 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9b3077bdd540eba971594d19c4b38b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:46 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:55972
I20260928 22:53:46.552244 16402 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9b3077bdd540eba971594d19c4b38b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:46 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:37766
I20260928 22:53:46.552280 16700 tablet_replica.cc:330] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c: stopping tablet replica
I20260928 22:53:46.552332 16699 tablet_replica.cc:330] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f: stopping tablet replica
I20260928 22:53:46.552336 16700 raft_consensus.cc:2229] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:46.552369 16700 raft_consensus.cc:2258] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:46.552377 16699 raft_consensus.cc:2229] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:53:46.552452 16699 raft_consensus.cc:2258] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:46.552451 16701 tablet_replica.cc:330] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e: stopping tablet replica
I20260928 22:53:46.552506 16701 raft_consensus.cc:2229] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:46.552538 16701 raft_consensus.cc:2258] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:46.552886 16701 ts_tablet_manager.cc:1905] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:46.553062 16699 ts_tablet_manager.cc:1905] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:46.554464 16700 ts_tablet_manager.cc:1905] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:46.554800 16701 ts_tablet_manager.cc:1918] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:53:46.554853 16701 log.cc:1196] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/fb9b3077bdd540eba971594d19c4b38b
I20260928 22:53:46.554908 16699 ts_tablet_manager.cc:1918] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:53:46.554973 16699 log.cc:1196] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/fb9b3077bdd540eba971594d19c4b38b
I20260928 22:53:46.555068 16701 ts_tablet_manager.cc:1939] T fb9b3077bdd540eba971594d19c4b38b P 0a69b854c27145c19fe37aaacfda117e: Deleting consensus metadata
I20260928 22:53:46.555195 16699 ts_tablet_manager.cc:1939] T fb9b3077bdd540eba971594d19c4b38b P 1535b5448d064ccda6f5ff6f99e10d9f: Deleting consensus metadata
I20260928 22:53:46.555496 16123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:53:46.555658 16123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:53:46.555682 16110 catalog_manager.cc:4841] TS 0a69b854c27145c19fe37aaacfda117e (127.14.14.131:40717): tablet fb9b3077bdd540eba971594d19c4b38b (table test-table [id=bde7f5ac8b9d41628cc01f87b319fd43]) successfully deleted
I20260928 22:53:46.556262 16109 catalog_manager.cc:4841] TS 1535b5448d064ccda6f5ff6f99e10d9f (127.14.14.130:45111): tablet fb9b3077bdd540eba971594d19c4b38b (table test-table [id=bde7f5ac8b9d41628cc01f87b319fd43]) successfully deleted
I20260928 22:53:46.556308 16700 ts_tablet_manager.cc:1918] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:53:46.556360 16700 log.cc:1196] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-0/wal/wals/fb9b3077bdd540eba971594d19c4b38b
I20260928 22:53:46.556578 16700 ts_tablet_manager.cc:1939] T fb9b3077bdd540eba971594d19c4b38b P d4fff65288a54c3fb142b3c7fdea7d7c: Deleting consensus metadata
I20260928 22:53:46.557061 16541 tablet_service.cc:1467] Processing CreateTablet for tablet 7a40a06d70a74558b15255b6e872dc40 (DEFAULT_TABLE table=test-table [id=3ae974e0875b418f8de0855d00636055]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:46.557098 16262 tablet_service.cc:1467] Processing CreateTablet for tablet 7a40a06d70a74558b15255b6e872dc40 (DEFAULT_TABLE table=test-table [id=3ae974e0875b418f8de0855d00636055]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:46.557089 16402 tablet_service.cc:1467] Processing CreateTablet for tablet 7a40a06d70a74558b15255b6e872dc40 (DEFAULT_TABLE table=test-table [id=3ae974e0875b418f8de0855d00636055]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:46.557170 16541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a40a06d70a74558b15255b6e872dc40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:46.557178 16262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a40a06d70a74558b15255b6e872dc40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:46.557190 16402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a40a06d70a74558b15255b6e872dc40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:46.558331 16710 tablet_bootstrap.cc:492] T 7a40a06d70a74558b15255b6e872dc40 P d4fff65288a54c3fb142b3c7fdea7d7c: Bootstrap starting.
I20260928 22:53:46.558382 16711 tablet_bootstrap.cc:492] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f: Bootstrap starting.
I20260928 22:53:46.558449 16712 tablet_bootstrap.cc:492] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e: Bootstrap starting.
I20260928 22:53:46.558791 16712 tablet_bootstrap.cc:654] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:46.558802 16711 tablet_bootstrap.cc:654] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:46.559007 16710 tablet_bootstrap.cc:654] T 7a40a06d70a74558b15255b6e872dc40 P d4fff65288a54c3fb142b3c7fdea7d7c: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:46.559368 16110 catalog_manager.cc:4841] TS d4fff65288a54c3fb142b3c7fdea7d7c (127.14.14.129:36883): tablet fb9b3077bdd540eba971594d19c4b38b (table test-table [id=bde7f5ac8b9d41628cc01f87b319fd43]) successfully deleted
I20260928 22:53:46.559669 16710 tablet_bootstrap.cc:492] T 7a40a06d70a74558b15255b6e872dc40 P d4fff65288a54c3fb142b3c7fdea7d7c: No bootstrap required, opened a new log
I20260928 22:53:46.559718 16710 ts_tablet_manager.cc:1397] T 7a40a06d70a74558b15255b6e872dc40 P d4fff65288a54c3fb142b3c7fdea7d7c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:46.559871 16712 tablet_bootstrap.cc:492] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e: No bootstrap required, opened a new log
I20260928 22:53:46.559866 16710 raft_consensus.cc:348] T 7a40a06d70a74558b15255b6e872dc40 P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.559928 16712 ts_tablet_manager.cc:1397] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:46.559958 16710 raft_consensus.cc:374] T 7a40a06d70a74558b15255b6e872dc40 P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:46.559986 16710 raft_consensus.cc:729] T 7a40a06d70a74558b15255b6e872dc40 P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4fff65288a54c3fb142b3c7fdea7d7c, State: Initialized, Role: FOLLOWER
I20260928 22:53:46.560041 16710 consensus_queue.cc:260] T 7a40a06d70a74558b15255b6e872dc40 P d4fff65288a54c3fb142b3c7fdea7d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.560034 16712 raft_consensus.cc:348] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.560125 16712 raft_consensus.cc:374] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:46.560129 16710 ts_tablet_manager.cc:1428] T 7a40a06d70a74558b15255b6e872dc40 P d4fff65288a54c3fb142b3c7fdea7d7c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:46.560148 16712 raft_consensus.cc:729] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69b854c27145c19fe37aaacfda117e, State: Initialized, Role: FOLLOWER
I20260928 22:53:46.560189 16712 consensus_queue.cc:260] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.560280 16712 ts_tablet_manager.cc:1428] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:46.560890 16711 tablet_bootstrap.cc:492] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f: No bootstrap required, opened a new log
I20260928 22:53:46.560941 16711 ts_tablet_manager.cc:1397] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 22:53:46.561125 16711 raft_consensus.cc:348] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.561206 16711 raft_consensus.cc:374] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:46.561231 16711 raft_consensus.cc:729] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1535b5448d064ccda6f5ff6f99e10d9f, State: Initialized, Role: FOLLOWER
I20260928 22:53:46.561272 16711 consensus_queue.cc:260] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.561394 16711 ts_tablet_manager.cc:1428] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:46.599467 16722 raft_consensus.cc:482] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:53:46.599575 16722 raft_consensus.cc:504] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.599736 16722 leader_election.cc:288] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4fff65288a54c3fb142b3c7fdea7d7c (127.14.14.129:36883), 0a69b854c27145c19fe37aaacfda117e (127.14.14.131:40717)
I20260928 22:53:46.600026 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a40a06d70a74558b15255b6e872dc40" candidate_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" is_pre_election: true
I20260928 22:53:46.600140 16265 raft_consensus.cc:2454] T 7a40a06d70a74558b15255b6e872dc40 P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1535b5448d064ccda6f5ff6f99e10d9f in term 0.
I20260928 22:53:46.600251 16562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a40a06d70a74558b15255b6e872dc40" candidate_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69b854c27145c19fe37aaacfda117e" is_pre_election: true
I20260928 22:53:46.600355 16562 raft_consensus.cc:2454] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1535b5448d064ccda6f5ff6f99e10d9f in term 0.
I20260928 22:53:46.600530 16354 leader_election.cc:302] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1535b5448d064ccda6f5ff6f99e10d9f, d4fff65288a54c3fb142b3c7fdea7d7c; no voters:
I20260928 22:53:46.600656 16722 raft_consensus.cc:2781] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:53:46.600715 16722 raft_consensus.cc:482] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:53:46.600746 16722 raft_consensus.cc:3037] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:46.601467 16722 raft_consensus.cc:504] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.601658 16722 leader_election.cc:288] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [CANDIDATE]: Term 1 election: Requested vote from peers d4fff65288a54c3fb142b3c7fdea7d7c (127.14.14.129:36883), 0a69b854c27145c19fe37aaacfda117e (127.14.14.131:40717)
I20260928 22:53:46.601794 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a40a06d70a74558b15255b6e872dc40" candidate_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c"
I20260928 22:53:46.601814 16562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a40a06d70a74558b15255b6e872dc40" candidate_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69b854c27145c19fe37aaacfda117e"
I20260928 22:53:46.601872 16265 raft_consensus.cc:3037] T 7a40a06d70a74558b15255b6e872dc40 P d4fff65288a54c3fb142b3c7fdea7d7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:46.601905 16562 raft_consensus.cc:3037] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:46.602589 16265 raft_consensus.cc:2454] T 7a40a06d70a74558b15255b6e872dc40 P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1535b5448d064ccda6f5ff6f99e10d9f in term 1.
I20260928 22:53:46.602756 16354 leader_election.cc:302] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1535b5448d064ccda6f5ff6f99e10d9f, d4fff65288a54c3fb142b3c7fdea7d7c; no voters:
I20260928 22:53:46.602891 16722 raft_consensus.cc:2781] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:46.602922 16562 raft_consensus.cc:2454] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1535b5448d064ccda6f5ff6f99e10d9f in term 1.
I20260928 22:53:46.602967 16722 raft_consensus.cc:686] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 LEADER]: Becoming Leader. State: Replica: 1535b5448d064ccda6f5ff6f99e10d9f, State: Running, Role: LEADER
I20260928 22:53:46.603020 16722 consensus_queue.cc:237] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } }
I20260928 22:53:46.603555 16123 catalog_manager.cc:5495] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1535b5448d064ccda6f5ff6f99e10d9f (127.14.14.130). New cstate: current_term: 1 leader_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1535b5448d064ccda6f5ff6f99e10d9f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 45111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 } health_report { overall_health: UNKNOWN } } }
I20260928 22:53:46.610996 16562 raft_consensus.cc:1261] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Refusing update from remote peer 1535b5448d064ccda6f5ff6f99e10d9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:46.611025 16265 raft_consensus.cc:1261] T 7a40a06d70a74558b15255b6e872dc40 P d4fff65288a54c3fb142b3c7fdea7d7c [term 1 FOLLOWER]: Refusing update from remote peer 1535b5448d064ccda6f5ff6f99e10d9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:46.611220 16717 consensus_queue.cc:1035] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4fff65288a54c3fb142b3c7fdea7d7c" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:46.611335 16717 consensus_queue.cc:1035] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69b854c27145c19fe37aaacfda117e" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 22:53:46.615986 16682 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 22:53:48.136332 16123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:48.136442 16123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:48.137256 16123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d88d3c8c71134d74a8c0e6c4ffb5d3bc: Sending DeleteTablet for 3 replicas of tablet 7a40a06d70a74558b15255b6e872dc40
I20260928 22:53:48.137498 16402 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a40a06d70a74558b15255b6e872dc40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:48 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:37766
I20260928 22:53:48.137519 16262 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a40a06d70a74558b15255b6e872dc40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:48 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:40070
I20260928 22:53:48.137552 16541 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a40a06d70a74558b15255b6e872dc40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:48 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:55972
I20260928 22:53:48.137768 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 16197
I20260928 22:53:48.141697 16766 tablet_replica.cc:330] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e: stopping tablet replica
I20260928 22:53:48.141777 16766 raft_consensus.cc:2229] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:48.141831 16766 raft_consensus.cc:2258] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:48.141886 16767 tablet_replica.cc:330] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f: stopping tablet replica
I20260928 22:53:48.141992 16767 raft_consensus.cc:2229] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:53:48.142064 16767 raft_consensus.cc:2258] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:48.142086 16766 ts_tablet_manager.cc:1905] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:48.142303 16767 ts_tablet_manager.cc:1905] T 7a40a06d70a74558b15255b6e872dc40 P 1535b5448d064ccda6f5ff6f99e10d9f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 22:53:48.143447 16110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:36883 (error 108)
I20260928 22:53:48.143671 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 16337
I20260928 22:53:48.144339 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:48.143966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:36883 (local address 127.0.0.1:40086)
0928 22:53:48.144246 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:48.144292 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:36883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":55,"threads_started":1}
W20260928 22:53:48.144431 16110 catalog_manager.cc:4584] TS d4fff65288a54c3fb142b3c7fdea7d7c (127.14.14.129:36883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a40a06d70a74558b15255b6e872dc40: Network error: Client connection negotiation failed: client connection to 127.14.14.129:36883: connect: Connection refused (error 111)
I20260928 22:53:48.144960 16766 ts_tablet_manager.cc:1918] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 22:53:48.145035 16766 log.cc:1196] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/7a40a06d70a74558b15255b6e872dc40
I20260928 22:53:48.145246 16766 ts_tablet_manager.cc:1939] T 7a40a06d70a74558b15255b6e872dc40 P 0a69b854c27145c19fe37aaacfda117e: Deleting consensus metadata
I20260928 22:53:48.145567 16110 catalog_manager.cc:4841] TS 0a69b854c27145c19fe37aaacfda117e (127.14.14.131:40717): tablet 7a40a06d70a74558b15255b6e872dc40 (table test-table [id=3ae974e0875b418f8de0855d00636055]) successfully deleted
I20260928 22:53:48.150600 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:48.150471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:45111 (local address 127.0.0.1:37782)
0928 22:53:48.150539 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:48.150570 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.130:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 22:53:48.150591 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 16477
I20260928 22:53:48.156422 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 16093
2026-09-28T22:53:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5712 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:48 dist-test-slave-kzfr krb5kdc[16772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:48 dist-test-slave-kzfr krb5kdc[16772](info): set up 2 sockets
Sep 28 22:53:48 dist-test-slave-kzfr krb5kdc[16772](info): commencing operation
krb5kdc: starting...
W20260928 22:53:49.225840 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:53:49 dist-test-slave-kzfr krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:53:49.345072 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:38643
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:35013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:38643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:49.424335 16785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:49.424561 16785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:49.424585 16785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:49.425885 16785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:49.425933 16785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:49.425948 16785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:49.425962 16785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:49.425974 16785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:49.427448 16785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:35013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:38643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:38643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:49.427647 16785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:49.427831 16785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:49.432999 16793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:49.433202 16791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:49.433182 16790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:49.433419 16785 server_base.cc:1034] running on GCE node
I20260928 22:53:49.433750 16785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:49.434051 16785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:49.435209 16785 hybrid_clock.cc:648] HybridClock initialized: now 1790636029435176 us; error 55 us; skew 500 ppm
Sep 28 22:53:49 dist-test-slave-kzfr krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636029, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:49.437762 16785 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:53:49.438236 16785 webserver.cc:460] Webserver started at http://127.14.14.190:39859/ using document root <none> and password file <none>
I20260928 22:53:49.438464 16785 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:49.438530 16785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:49.438679 16785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:49.439764 16785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "ef4c69a40b8a43668d150a7e8e9ffdf8"
format_stamp: "Formatted at 2026-09-28 22:53:49 on dist-test-slave-kzfr"
server_key: "f28d4441852a37b95e133c4eff9f4d8c"
server_key_iv: "391ef7486cd4bf61227e9514007ae9ed"
server_key_version: "encryptionkey@0"
I20260928 22:53:49.440126 16785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "ef4c69a40b8a43668d150a7e8e9ffdf8"
format_stamp: "Formatted at 2026-09-28 22:53:49 on dist-test-slave-kzfr"
server_key: "f28d4441852a37b95e133c4eff9f4d8c"
server_key_iv: "391ef7486cd4bf61227e9514007ae9ed"
server_key_version: "encryptionkey@0"
I20260928 22:53:49.441509 16785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:53:49.442325 16800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:49.442632 16785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:49.442754 16785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "ef4c69a40b8a43668d150a7e8e9ffdf8"
format_stamp: "Formatted at 2026-09-28 22:53:49 on dist-test-slave-kzfr"
server_key: "f28d4441852a37b95e133c4eff9f4d8c"
server_key_iv: "391ef7486cd4bf61227e9514007ae9ed"
server_key_version: "encryptionkey@0"
I20260928 22:53:49.442831 16785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:49.466900 16785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:49.468824 16785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:49.468997 16785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:49.473474 16785 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:38643
I20260928 22:53:49.473546 16852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:38643 every 8 connection(s)
I20260928 22:53:49.473976 16785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:49.474984 16853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:49.477501 16853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8: Bootstrap starting.
I20260928 22:53:49.478185 16853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:49.478530 16853 log.cc:824] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:49.479255 16853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8: No bootstrap required, opened a new log
I20260928 22:53:49.479903 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 16785
I20260928 22:53:49.480007 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:53:49.480182 14394 external_mini_cluster.cc:1428] Setting key d8a76e6baf001d9374391664d5b567a6
I20260928 22:53:49.481536 16853 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4c69a40b8a43668d150a7e8e9ffdf8" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38643 } }
I20260928 22:53:49.481750 16853 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:49.481789 16853 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef4c69a40b8a43668d150a7e8e9ffdf8, State: Initialized, Role: FOLLOWER
I20260928 22:53:49.481941 16853 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4c69a40b8a43668d150a7e8e9ffdf8" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38643 } }
I20260928 22:53:49.482028 16853 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:49.482051 16853 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:49.482117 16853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:49.483222 16853 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4c69a40b8a43668d150a7e8e9ffdf8" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38643 } }
I20260928 22:53:49.483326 16853 leader_election.cc:302] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef4c69a40b8a43668d150a7e8e9ffdf8; no voters:
I20260928 22:53:49.483479 16853 leader_election.cc:288] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:49.483544 16858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:49.483677 16858 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [term 1 LEADER]: Becoming Leader. State: Replica: ef4c69a40b8a43668d150a7e8e9ffdf8, State: Running, Role: LEADER
I20260928 22:53:49.483743 16853 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:49.483779 16858 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4c69a40b8a43668d150a7e8e9ffdf8" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38643 } }
I20260928 22:53:49.484125 16858 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef4c69a40b8a43668d150a7e8e9ffdf8. Latest consensus state: current_term: 1 leader_uuid: "ef4c69a40b8a43668d150a7e8e9ffdf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4c69a40b8a43668d150a7e8e9ffdf8" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38643 } } }
I20260928 22:53:49.484194 16858 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:49.484511 16860 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef4c69a40b8a43668d150a7e8e9ffdf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4c69a40b8a43668d150a7e8e9ffdf8" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38643 } } }
I20260928 22:53:49.484620 16860 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:49.484933 16865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:49.485445 16865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:49.487196 16865 catalog_manager.cc:1347] Generated new cluster ID: 896ab197cecd45029e7565bddd405b79
I20260928 22:53:49.487253 16865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:53:49 dist-test-slave-kzfr krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:49.492029 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:49.480794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58452 (local address 127.14.14.190:38643)
0928 22:53:49.481007 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:49.481017 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:49.481061 (+ 44us) server_negotiation.cc:407] Connection header received
0928 22:53:49.481494 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:49.481503 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:49.481569 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:49.481733 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:49.482497 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:49.483944 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:49.485023 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:49.485676 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:49.485782 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:49.488388 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:49.488392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:49.488392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:49.488404 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:49.490147 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:49.490589 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:49.490591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:49.490592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:49.490707 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:49.490920 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:49.490920 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:49.490921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:49.491157 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:49.491735 (+ 578us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:49.491791 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 22:53:49.491891 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260928 22:53:49.505496 16865 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:49.506091 16865 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:49.509109 16865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef4c69a40b8a43668d150a7e8e9ffdf8: Generated new TSK 0
I20260928 22:53:49.509305 16865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:53:49.541265 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38643
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:35013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:49.626986 16879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:49.627213 16879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:49.627251 16879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:49.628388 16879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:49.628459 16879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:49.628504 16879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:53:49.630004 16879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:35013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:49.630268 16879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:49.630498 16879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:49.635524 16887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:49.635509 16885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:49.635562 16879 server_base.cc:1034] running on GCE node
W20260928 22:53:49.635505 16884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:49.635888 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:49.636144 16879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:49.637306 16879 hybrid_clock.cc:648] HybridClock initialized: now 1790636029637299 us; error 37 us; skew 500 ppm
Sep 28 22:53:49 dist-test-slave-kzfr krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636029, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:49.639729 16879 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:53:49.640202 16879 webserver.cc:460] Webserver started at http://127.14.14.129:41017/ using document root <none> and password file <none>
I20260928 22:53:49.640442 16879 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:49.640511 16879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:49.640674 16879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:49.641782 16879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "a0cf4db8383d489090343f39003241bf"
format_stamp: "Formatted at 2026-09-28 22:53:49 on dist-test-slave-kzfr"
server_key: "53ae5c8b3e5b8a4f9ae47f747266b571"
server_key_iv: "3403b1be33ffbdb88806094755198053"
server_key_version: "encryptionkey@0"
I20260928 22:53:49.642150 16879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "a0cf4db8383d489090343f39003241bf"
format_stamp: "Formatted at 2026-09-28 22:53:49 on dist-test-slave-kzfr"
server_key: "53ae5c8b3e5b8a4f9ae47f747266b571"
server_key_iv: "3403b1be33ffbdb88806094755198053"
server_key_version: "encryptionkey@0"
I20260928 22:53:49.643435 16879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:53:49.644138 16894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:49.644310 16879 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:49.644407 16879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "a0cf4db8383d489090343f39003241bf"
format_stamp: "Formatted at 2026-09-28 22:53:49 on dist-test-slave-kzfr"
server_key: "53ae5c8b3e5b8a4f9ae47f747266b571"
server_key_iv: "3403b1be33ffbdb88806094755198053"
server_key_version: "encryptionkey@0"
I20260928 22:53:49.644485 16879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:49.672829 16879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:49.674882 16879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:49.675052 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:49.675287 16879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:49.675596 16879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:49.675652 16879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:49.675695 16879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:49.675724 16879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:49.681921 16879 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:35987
I20260928 22:53:49.681982 17007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:35987 every 8 connection(s)
I20260928 22:53:49.682333 16879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:53:49.686793 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 16879
I20260928 22:53:49.686897 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:53:49.687084 14394 external_mini_cluster.cc:1428] Setting key 798476a11471a065b0ce555e584c9f5b
Sep 28 22:53:49 dist-test-slave-kzfr krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636029, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:49.692694 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:49.683098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:49797 (local address 127.14.14.190:38643)
0928 22:53:49.683172 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:49.683174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:49.683365 (+ 191us) server_negotiation.cc:407] Connection header received
0928 22:53:49.683939 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:49.683940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:49.683957 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:49.684013 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:49.685133 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:49.685771 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:49.686769 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:49.687067 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:49.687153 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:49.689750 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:49.689751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:49.689752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:49.689763 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:49.691282 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:49.691715 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:49.691716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:49.691717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:49.691786 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:49.692048 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:49.692049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:49.692050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:49.692167 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:49.692242 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:49.692518 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 22:53:49.692604 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 22:53:49.692895 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:49.683059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38643 (local address 127.14.14.129:49797)
0928 22:53:49.683239 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:49.683274 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:49.683823 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:49.684085 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:49.684098 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:49.684819 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:49.685020 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:49.685025 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:49.685861 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:49.685862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:49.686675 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:49.686677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:49.687189 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:49.687190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:49.687364 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:49.687978 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:49.687987 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:49.689661 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:49.691390 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:49.691397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:49.691399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:49.691629 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:49.691865 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:49.691865 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:49.691866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:49.691975 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:49.692241 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:49.692248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:49.692387 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 22:53:49.692496 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 22:53:49.692652 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260928 22:53:49.693395 17008 heartbeater.cc:347] Connected to a master server at 127.14.14.190:38643
I20260928 22:53:49.693542 17008 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:49.693756 17008 heartbeater.cc:511] Master 127.14.14.190:38643 requested a full tablet report, sending...
I20260928 22:53:49.694300 16817 ts_manager.cc:194] Registered new tserver with Master: a0cf4db8383d489090343f39003241bf (127.14.14.129:35987)
I20260928 22:53:49.694908 16817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:49797
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:53:49.712741 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38643
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:35013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:49.792469 17013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:49.792675 17013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:49.792697 17013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:49.793809 17013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:49.793845 17013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:49.793871 17013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:53:49.795207 17013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:35013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:49.795396 17013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:49.795565 17013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:49.801021 17021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:49.801154 17013 server_base.cc:1034] running on GCE node
W20260928 22:53:49.801013 17019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:49.801013 17018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:49.801569 17013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:49.801898 17013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:49.803072 17013 hybrid_clock.cc:648] HybridClock initialized: now 1790636029803048 us; error 38 us; skew 500 ppm
Sep 28 22:53:49 dist-test-slave-kzfr krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636029, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:49.807150 17013 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:53:49.807770 17013 webserver.cc:460] Webserver started at http://127.14.14.130:35917/ using document root <none> and password file <none>
I20260928 22:53:49.808072 17013 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:49.808136 17013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:49.808276 17013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:49.809996 17013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "38e878af06444fcf8319c96f6b99fe56"
format_stamp: "Formatted at 2026-09-28 22:53:49 on dist-test-slave-kzfr"
server_key: "bad9fcecab83fb2372c4dcd6f8e72372"
server_key_iv: "ea5db0095f00ef4211e5f0e65b14652c"
server_key_version: "encryptionkey@0"
I20260928 22:53:49.810462 17013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "38e878af06444fcf8319c96f6b99fe56"
format_stamp: "Formatted at 2026-09-28 22:53:49 on dist-test-slave-kzfr"
server_key: "bad9fcecab83fb2372c4dcd6f8e72372"
server_key_iv: "ea5db0095f00ef4211e5f0e65b14652c"
server_key_version: "encryptionkey@0"
I20260928 22:53:49.812470 17013 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 22:53:49.813328 17028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:49.813522 17013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:49.813635 17013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "38e878af06444fcf8319c96f6b99fe56"
format_stamp: "Formatted at 2026-09-28 22:53:49 on dist-test-slave-kzfr"
server_key: "bad9fcecab83fb2372c4dcd6f8e72372"
server_key_iv: "ea5db0095f00ef4211e5f0e65b14652c"
server_key_version: "encryptionkey@0"
I20260928 22:53:49.813727 17013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:49.835644 17013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:49.838057 17013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:49.838225 17013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:49.838481 17013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:49.838864 17013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:49.838918 17013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:49.838959 17013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:49.838986 17013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:49.844884 17013 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:44521
I20260928 22:53:49.844928 17141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:44521 every 8 connection(s)
I20260928 22:53:49.845271 17013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:53:49.850343 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 17013
I20260928 22:53:49.850441 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:53:49.850628 14394 external_mini_cluster.cc:1428] Setting key 90f3d6c681a9d10958eef6fcd2cd0958
Sep 28 22:53:49 dist-test-slave-kzfr krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636029, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:49.854887 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:49.845988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:52459 (local address 127.14.14.190:38643)
0928 22:53:49.846033 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:49.846036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:49.846277 (+ 241us) server_negotiation.cc:407] Connection header received
0928 22:53:49.846790 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:49.846790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:49.846801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:49.846858 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:49.847819 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:49.848311 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:49.849321 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:49.849497 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:49.849548 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:49.851969 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:49.851971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:49.851971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:49.851979 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:49.853348 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:49.853867 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:49.853868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:49.853868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:49.853923 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:49.854196 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:49.854196 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:49.854196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:49.854307 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:49.854359 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:49.854680 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 22:53:49.854747 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:53:49.855121 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:49.845985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38643 (local address 127.14.14.130:52459)
0928 22:53:49.846161 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:49.846206 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:49.846688 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:49.846911 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:49.846915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:49.847535 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:49.847728 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:49.847734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:49.848430 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:49.848431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:49.849248 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:49.849250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:49.849566 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:49.849567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:49.849727 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:49.850205 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:49.850215 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:49.851872 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:49.853448 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:49.853455 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:49.853457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:49.853780 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:49.853989 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:49.853990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:49.853991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:49.854134 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:49.854372 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:49.854378 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:49.854544 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 22:53:49.854663 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 22:53:49.854842 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260928 22:53:49.855700 17142 heartbeater.cc:347] Connected to a master server at 127.14.14.190:38643
I20260928 22:53:49.855834 17142 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:49.856010 17142 heartbeater.cc:511] Master 127.14.14.190:38643 requested a full tablet report, sending...
I20260928 22:53:49.856456 16817 ts_manager.cc:194] Registered new tserver with Master: 38e878af06444fcf8319c96f6b99fe56 (127.14.14.130:44521)
I20260928 22:53:49.856999 16817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:52459
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:53:49.876292 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38643
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:35013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:49.956175 17147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:49.956377 17147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:49.956399 17147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:49.957563 17147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:49.957623 17147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:49.957649 17147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:53:49.959107 17147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:35013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:49.959340 17147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:49.959576 17147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:49.964567 17155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:49.964709 17152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:49.964900 17147 server_base.cc:1034] running on GCE node
W20260928 22:53:49.964999 17153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:49.965263 17147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:49.965514 17147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:49.966672 17147 hybrid_clock.cc:648] HybridClock initialized: now 1790636029966643 us; error 43 us; skew 500 ppm
Sep 28 22:53:49 dist-test-slave-kzfr krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636029, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:49.969115 17147 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:53:49.969621 17147 webserver.cc:460] Webserver started at http://127.14.14.131:43467/ using document root <none> and password file <none>
I20260928 22:53:49.969854 17147 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:49.969920 17147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:49.970075 17147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:49.971079 17147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "6e59d528c8e04d7598d77e24e28f6806"
format_stamp: "Formatted at 2026-09-28 22:53:49 on dist-test-slave-kzfr"
server_key: "5adb3d4f8e0ee7a24a03bf83ca852529"
server_key_iv: "2e027b9044308deb2171ae78060cec11"
server_key_version: "encryptionkey@0"
I20260928 22:53:49.971441 17147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "6e59d528c8e04d7598d77e24e28f6806"
format_stamp: "Formatted at 2026-09-28 22:53:49 on dist-test-slave-kzfr"
server_key: "5adb3d4f8e0ee7a24a03bf83ca852529"
server_key_iv: "2e027b9044308deb2171ae78060cec11"
server_key_version: "encryptionkey@0"
I20260928 22:53:49.972689 17147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:49.973390 17162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:49.973647 17147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:49.973764 17147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "6e59d528c8e04d7598d77e24e28f6806"
format_stamp: "Formatted at 2026-09-28 22:53:49 on dist-test-slave-kzfr"
server_key: "5adb3d4f8e0ee7a24a03bf83ca852529"
server_key_iv: "2e027b9044308deb2171ae78060cec11"
server_key_version: "encryptionkey@0"
I20260928 22:53:49.973850 17147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:49.993022 17147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:49.995322 17147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:49.995482 17147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:49.995704 17147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:49.996025 17147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:49.996101 17147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:49.996141 17147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:49.996170 17147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:50.002332 17147 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:35277
I20260928 22:53:50.002378 17275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:35277 every 8 connection(s)
I20260928 22:53:50.002687 17147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:53:50 dist-test-slave-kzfr krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636029, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:50.011212 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 17147
I20260928 22:53:50.011304 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:53:50.011475 14394 external_mini_cluster.cc:1428] Setting key 70f11765a424cd88602995a9e0af0f03
I20260928 22:53:50.011797 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:50.003421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:51793 (local address 127.14.14.190:38643)
0928 22:53:50.003499 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:50.003502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:50.003703 (+ 201us) server_negotiation.cc:407] Connection header received
0928 22:53:50.004169 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:50.004170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:50.004182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:50.004236 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:50.005141 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:50.005764 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:50.006687 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:50.006869 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:50.006923 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:50.009094 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:50.009096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:50.009096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:50.009116 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:50.010444 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:50.010794 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:50.010795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:50.010796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:50.010844 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:50.011086 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:50.011087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:50.011087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:50.011242 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:50.011318 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:50.011613 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 22:53:50.011696 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 22:53:50.011974 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:50.003365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38643 (local address 127.14.14.131:51793)
0928 22:53:50.003546 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:50.003582 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:50.004081 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:50.004291 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:50.004294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:50.004851 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:50.005043 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:50.005048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:50.005839 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:50.005839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:50.006605 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:50.006606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:50.006939 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:50.006940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:50.007036 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:50.007564 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:50.007572 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:50.009002 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:50.010513 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:50.010519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:50.010521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:50.010729 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:50.010900 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:50.010900 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:50.010900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:50.010992 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:50.011317 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:50.011322 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:50.011465 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 22:53:50.011586 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 22:53:50.011739 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260928 22:53:50.012424 17276 heartbeater.cc:347] Connected to a master server at 127.14.14.190:38643
I20260928 22:53:50.012539 17276 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:50.012710 17276 heartbeater.cc:511] Master 127.14.14.190:38643 requested a full tablet report, sending...
I20260928 22:53:50.013056 16817 ts_manager.cc:194] Registered new tserver with Master: 6e59d528c8e04d7598d77e24e28f6806 (127.14.14.131:35277)
I20260928 22:53:50.013381 16817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:51793
I20260928 22:53:50.023152 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 22:53:50.033504 16857 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:50.032703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58462 (local address 127.14.14.190:38643)
0928 22:53:50.032771 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:50.032772 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:50.032871 (+ 99us) server_negotiation.cc:407] Connection header received
0928 22:53:50.032910 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:50.032911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:50.032920 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:50.032973 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:50.033454 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58462: BlockingRecv error: recv got EOF from 127.0.0.1:58462 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:53:50.034435 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 16879
I20260928 22:53:50.038833 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 17013
I20260928 22:53:50.043169 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 17147
I20260928 22:53:50.047518 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 16785
2026-09-28T22:53:50Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1889 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:50 dist-test-slave-kzfr krb5kdc[17288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:50 dist-test-slave-kzfr krb5kdc[17288](info): set up 2 sockets
Sep 28 22:53:50 dist-test-slave-kzfr krb5kdc[17288](info): commencing operation
krb5kdc: starting...
W20260928 22:53:51.077777 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:53:51 dist-test-slave-kzfr krb5kdc[17288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:53:51.196305 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:33841
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:33841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:51.278669 17301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:51.278913 17301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:51.278951 17301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:51.280122 17301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:51.280189 17301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:51.280215 17301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:51.280246 17301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:51.280275 17301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:51.281857 17301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:33841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:33841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:51.282130 17301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:51.282351 17301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:51.287317 17309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:51.287297 17306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:51.287333 17307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:51.287719 17301 server_base.cc:1034] running on GCE node
I20260928 22:53:51.287956 17301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:51.288286 17301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:51.289456 17301 hybrid_clock.cc:648] HybridClock initialized: now 1790636031289445 us; error 32 us; skew 500 ppm
Sep 28 22:53:51 dist-test-slave-kzfr krb5kdc[17288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636031, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:51.292083 17301 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:53:51.292531 17301 webserver.cc:460] Webserver started at http://127.14.14.190:35521/ using document root <none> and password file <none>
I20260928 22:53:51.292763 17301 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:51.292832 17301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:51.292987 17301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:51.294083 17301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "1b72527ebbd647d3996e628079e7f162"
format_stamp: "Formatted at 2026-09-28 22:53:51 on dist-test-slave-kzfr"
server_key: "c6298ae7245ceef061e2b965b8fefc73"
server_key_iv: "fa671fe5dc1d4fa54cc96bf12d914aff"
server_key_version: "encryptionkey@0"
I20260928 22:53:51.294451 17301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "1b72527ebbd647d3996e628079e7f162"
format_stamp: "Formatted at 2026-09-28 22:53:51 on dist-test-slave-kzfr"
server_key: "c6298ae7245ceef061e2b965b8fefc73"
server_key_iv: "fa671fe5dc1d4fa54cc96bf12d914aff"
server_key_version: "encryptionkey@0"
I20260928 22:53:51.295758 17301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:53:51.296473 17316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:51.296682 17301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:51.296787 17301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "1b72527ebbd647d3996e628079e7f162"
format_stamp: "Formatted at 2026-09-28 22:53:51 on dist-test-slave-kzfr"
server_key: "c6298ae7245ceef061e2b965b8fefc73"
server_key_iv: "fa671fe5dc1d4fa54cc96bf12d914aff"
server_key_version: "encryptionkey@0"
I20260928 22:53:51.296867 17301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:51.314419 17301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:51.316807 17301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:51.316975 17301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:51.321318 17301 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:33841
I20260928 22:53:51.321375 17368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:33841 every 8 connection(s)
I20260928 22:53:51.321887 17301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:51.322538 17369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:51.324815 17369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162: Bootstrap starting.
I20260928 22:53:51.325395 17369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:51.325726 17369 log.cc:824] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:51.326401 17369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162: No bootstrap required, opened a new log
I20260928 22:53:51.327942 17369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b72527ebbd647d3996e628079e7f162" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33841 } }
I20260928 22:53:51.328089 17369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:51.328114 17369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b72527ebbd647d3996e628079e7f162, State: Initialized, Role: FOLLOWER
I20260928 22:53:51.328291 17369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b72527ebbd647d3996e628079e7f162" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33841 } }
I20260928 22:53:51.328352 17369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:51.328405 17369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:51.328452 17369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:51.329321 17369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b72527ebbd647d3996e628079e7f162" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33841 } }
I20260928 22:53:51.329432 17369 leader_election.cc:302] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b72527ebbd647d3996e628079e7f162; no voters:
I20260928 22:53:51.329654 17369 leader_election.cc:288] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:51.329686 17372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:51.329849 17372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [term 1 LEADER]: Becoming Leader. State: Replica: 1b72527ebbd647d3996e628079e7f162, State: Running, Role: LEADER
I20260928 22:53:51.329941 17372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b72527ebbd647d3996e628079e7f162" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33841 } }
I20260928 22:53:51.330008 17369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:51.330288 17373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b72527ebbd647d3996e628079e7f162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b72527ebbd647d3996e628079e7f162" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33841 } } }
I20260928 22:53:51.330308 17374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b72527ebbd647d3996e628079e7f162. Latest consensus state: current_term: 1 leader_uuid: "1b72527ebbd647d3996e628079e7f162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b72527ebbd647d3996e628079e7f162" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33841 } } }
I20260928 22:53:51.330369 17373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:51.330379 17374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:51.330592 17376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:51.331099 17376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:51.331256 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 17301
I20260928 22:53:51.331351 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:53:51.331530 14394 external_mini_cluster.cc:1428] Setting key ec03a0cd0e76c4da4bc8934f92d4d659
I20260928 22:53:51.332712 17376 catalog_manager.cc:1347] Generated new cluster ID: 32a0f66025e84dfab6a20946e702c789
I20260928 22:53:51.332769 17376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:53:51 dist-test-slave-kzfr krb5kdc[17288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:51.341414 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:51.332128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.14.14.190:33841)
0928 22:53:51.332510 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:51.332517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:51.332539 (+ 22us) server_negotiation.cc:407] Connection header received
0928 22:53:51.332676 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:51.332683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:51.332736 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:51.332916 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:51.333532 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:51.334488 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:51.335230 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:51.335658 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:51.335785 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:51.338260 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:51.338266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:51.338267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:51.338282 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:51.339797 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:51.340157 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:51.340160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:51.340160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:51.340249 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:51.340472 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:51.340473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:51.340473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:51.340801 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:51.340972 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:51.341080 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 22:53:51.341219 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":64,"threads_started":1}
I20260928 22:53:51.342392 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:51.349247 17376 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:51.349965 17376 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:51.357142 17376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b72527ebbd647d3996e628079e7f162: Generated new TSK 0
I20260928 22:53:51.357326 17376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:51.368400 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:51.364530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.14.14.190:33841)
0928 22:53:51.364578 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:51.364580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:51.364684 (+ 104us) server_negotiation.cc:407] Connection header received
0928 22:53:51.364755 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:51.364756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:51.364766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:51.364816 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:51.365239 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:51.365842 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:51.366908 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:51.367115 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:51.367167 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:51.367913 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:51.367914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:51.367914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:51.368101 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:51.368116 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:51.368122 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:51.368169 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:51.368252 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 22:53:51.368289 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:53:51.370973 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 17301
2026-09-28T22:53:51Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1320 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:51 dist-test-slave-kzfr krb5kdc[17402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:51 dist-test-slave-kzfr krb5kdc[17402](info): set up 2 sockets
Sep 28 22:53:51 dist-test-slave-kzfr krb5kdc[17402](info): commencing operation
krb5kdc: starting...
W20260928 22:53:52.405817 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:53:52 dist-test-slave-kzfr krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:53:52.524541 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:46023
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:46023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:52.603709 17415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:52.603904 17415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:52.603925 17415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:52.605057 17415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:52.605101 17415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:52.605115 17415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:52.605129 17415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:52.605140 17415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:52.606652 17415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:46023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:46023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:52.606899 17415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:52.607134 17415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:52.612061 17420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:52.612187 17415 server_base.cc:1034] running on GCE node
W20260928 22:53:52.612056 17421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:52.612058 17423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:52.612556 17415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:52.612787 17415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:52.613922 17415 hybrid_clock.cc:648] HybridClock initialized: now 1790636032613909 us; error 29 us; skew 500 ppm
Sep 28 22:53:52 dist-test-slave-kzfr krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:52.616209 17415 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:53:52.616623 17415 webserver.cc:460] Webserver started at http://127.14.14.190:46049/ using document root <none> and password file <none>
I20260928 22:53:52.616856 17415 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:52.616901 17415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:52.617067 17415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:52.618189 17415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "b5a4b5b29e12471682a2756f89bb006d"
format_stamp: "Formatted at 2026-09-28 22:53:52 on dist-test-slave-kzfr"
server_key: "565e7bddfaa3e5f9c571fd62a3bfe0b8"
server_key_iv: "73fbea61279ac2808460769b585e2204"
server_key_version: "encryptionkey@0"
I20260928 22:53:52.618625 17415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "b5a4b5b29e12471682a2756f89bb006d"
format_stamp: "Formatted at 2026-09-28 22:53:52 on dist-test-slave-kzfr"
server_key: "565e7bddfaa3e5f9c571fd62a3bfe0b8"
server_key_iv: "73fbea61279ac2808460769b585e2204"
server_key_version: "encryptionkey@0"
I20260928 22:53:52.619915 17415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:53:52.620618 17430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:52.620781 17415 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:52.620884 17415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "b5a4b5b29e12471682a2756f89bb006d"
format_stamp: "Formatted at 2026-09-28 22:53:52 on dist-test-slave-kzfr"
server_key: "565e7bddfaa3e5f9c571fd62a3bfe0b8"
server_key_iv: "73fbea61279ac2808460769b585e2204"
server_key_version: "encryptionkey@0"
I20260928 22:53:52.620960 17415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:52.632046 17415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:52.633939 17415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:52.634092 17415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:52.637748 17415 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:46023
I20260928 22:53:52.637799 17482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:46023 every 8 connection(s)
I20260928 22:53:52.638145 17415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:52.638846 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 17415
I20260928 22:53:52.638935 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:53:52.639142 14394 external_mini_cluster.cc:1428] Setting key 7c7451f7d089cfd3ef5bd7488995ca92
I20260928 22:53:52.639194 17483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:52.642501 17483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d: Bootstrap starting.
I20260928 22:53:52.643105 17483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:52.643424 17483 log.cc:824] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:52.644125 17483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d: No bootstrap required, opened a new log
Sep 28 22:53:52 dist-test-slave-kzfr krb5kdc[17402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:52.646080 17483 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a4b5b29e12471682a2756f89bb006d" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46023 } }
I20260928 22:53:52.646245 17483 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:52.646286 17483 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a4b5b29e12471682a2756f89bb006d, State: Initialized, Role: FOLLOWER
I20260928 22:53:52.646425 17483 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a4b5b29e12471682a2756f89bb006d" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46023 } }
I20260928 22:53:52.646526 17483 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:52.646566 17483 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:52.646620 17483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:52.647138 17483 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a4b5b29e12471682a2756f89bb006d" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46023 } }
I20260928 22:53:52.647220 17483 leader_election.cc:302] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5a4b5b29e12471682a2756f89bb006d; no voters:
I20260928 22:53:52.647401 17483 leader_election.cc:288] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:52.647430 17488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:52.647545 17488 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [term 1 LEADER]: Becoming Leader. State: Replica: b5a4b5b29e12471682a2756f89bb006d, State: Running, Role: LEADER
I20260928 22:53:52.647624 17488 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a4b5b29e12471682a2756f89bb006d" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46023 } }
I20260928 22:53:52.647670 17483 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:52.647976 17489 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5a4b5b29e12471682a2756f89bb006d. Latest consensus state: current_term: 1 leader_uuid: "b5a4b5b29e12471682a2756f89bb006d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a4b5b29e12471682a2756f89bb006d" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46023 } } }
I20260928 22:53:52.648464 17489 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:52.647989 17490 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5a4b5b29e12471682a2756f89bb006d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a4b5b29e12471682a2756f89bb006d" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46023 } } }
I20260928 22:53:52.648685 17493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:52.648698 17490 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:52.649300 17493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:52.650779 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:52.639726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49720 (local address 127.14.14.190:46023)
0928 22:53:52.639903 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:52.639913 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:52.639945 (+ 32us) server_negotiation.cc:407] Connection header received
0928 22:53:52.640375 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:52.640383 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:52.640443 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:52.640603 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:52.641194 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:52.642594 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:52.643381 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:52.643927 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:52.644035 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:52.646642 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:52.646646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:52.646646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:52.646657 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:52.648722 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:52.649150 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:52.649153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:52.649154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:52.649254 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:52.649480 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:52.649481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:52.649481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:52.649890 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:52.650388 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:52.650456 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 22:53:52.650592 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260928 22:53:52.650924 17493 catalog_manager.cc:1347] Generated new cluster ID: 32cd934a121d4555918db0bb30c2d38e
I20260928 22:53:52.651095 17493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:52.662639 17493 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:52.663141 17493 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:52.667882 17493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5a4b5b29e12471682a2756f89bb006d: Generated new TSK 0
I20260928 22:53:52.668053 17493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:53:52.700474 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46023
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:52.788311 17509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:52.788506 17509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:52.788530 17509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:52.789844 17509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:52.789889 17509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:52.789916 17509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:53:52.791389 17509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:52.791639 17509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:52.791893 17509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:52.797070 17515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:52.797117 17509 server_base.cc:1034] running on GCE node
W20260928 22:53:52.797070 17517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:52.797111 17514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:52.797501 17509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:52.797780 17509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:52.798923 17509 hybrid_clock.cc:648] HybridClock initialized: now 1790636032798911 us; error 29 us; skew 500 ppm
Sep 28 22:53:52 dist-test-slave-kzfr krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:52.801406 17509 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:53:52.801910 17509 webserver.cc:460] Webserver started at http://127.14.14.129:33701/ using document root <none> and password file <none>
I20260928 22:53:52.802111 17509 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:52.802155 17509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:52.802244 17509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:52.803306 17509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "75daec32e8f846d38c10fd8e7ede98b8"
format_stamp: "Formatted at 2026-09-28 22:53:52 on dist-test-slave-kzfr"
server_key: "d0c24a3ae11663f5c78fb4f20f8079c1"
server_key_iv: "dcdcc735da5ee166f805314c26e29e5a"
server_key_version: "encryptionkey@0"
I20260928 22:53:52.803611 17509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "75daec32e8f846d38c10fd8e7ede98b8"
format_stamp: "Formatted at 2026-09-28 22:53:52 on dist-test-slave-kzfr"
server_key: "d0c24a3ae11663f5c78fb4f20f8079c1"
server_key_iv: "dcdcc735da5ee166f805314c26e29e5a"
server_key_version: "encryptionkey@0"
I20260928 22:53:52.804877 17509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:53:52.805574 17524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:52.805790 17509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:53:52.805904 17509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "75daec32e8f846d38c10fd8e7ede98b8"
format_stamp: "Formatted at 2026-09-28 22:53:52 on dist-test-slave-kzfr"
server_key: "d0c24a3ae11663f5c78fb4f20f8079c1"
server_key_iv: "dcdcc735da5ee166f805314c26e29e5a"
server_key_version: "encryptionkey@0"
I20260928 22:53:52.805997 17509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:52.819643 17509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:52.821630 17509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:52.821816 17509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:52.822069 17509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:52.822400 17509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:52.822459 17509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:52.822505 17509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:52.822531 17509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:52.828680 17509 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:44271
I20260928 22:53:52.828720 17637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:44271 every 8 connection(s)
I20260928 22:53:52.829061 17509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:53:52 dist-test-slave-kzfr krb5kdc[17402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:52.835747 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 17509
I20260928 22:53:52.835875 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:53:52.836063 14394 external_mini_cluster.cc:1428] Setting key fae86010cb3c49dfeda59ed825aa53eb
I20260928 22:53:52.838773 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:52.829853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:50043 (local address 127.14.14.190:46023)
0928 22:53:52.829932 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:52.829934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:52.830168 (+ 234us) server_negotiation.cc:407] Connection header received
0928 22:53:52.830631 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:52.830632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:52.830641 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:52.830695 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:52.831595 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:52.832091 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:52.833099 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:52.833283 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:52.833331 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:52.835454 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:52.835455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:52.835455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:52.835462 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:52.837182 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:52.837716 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:52.837717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:52.837718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:52.837770 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:52.838048 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:52.838049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:52.838050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:52.838148 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:52.838222 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:52.838559 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 22:53:52.838652 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 22:53:52.838809 17640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:52.829838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:46023 (local address 127.14.14.129:50043)
0928 22:53:52.830047 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:52.830083 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:52.830553 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:52.830752 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:52.830755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:52.831317 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:52.831518 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:52.831522 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:52.832167 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:52.832168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:52.833027 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:52.833029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:52.833351 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:52.833351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:52.833444 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:52.834004 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:52.834012 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:52.835369 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:52.837263 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:52.837270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:52.837272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:52.837641 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:52.837838 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:52.837839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:52.837839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:52.837984 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:52.838217 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:52.838223 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:52.838400 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 22:53:52.838510 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 22:53:52.838629 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":136,"threads_started":1}
I20260928 22:53:52.839684 17638 heartbeater.cc:347] Connected to a master server at 127.14.14.190:46023
I20260928 22:53:52.839789 17638 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:52.839987 17638 heartbeater.cc:511] Master 127.14.14.190:46023 requested a full tablet report, sending...
I20260928 22:53:52.840514 17447 ts_manager.cc:194] Registered new tserver with Master: 75daec32e8f846d38c10fd8e7ede98b8 (127.14.14.129:44271)
I20260928 22:53:52.841224 17447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:50043
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:53:52.864912 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46023
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:52.946991 17643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:52.947237 17643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:52.947278 17643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:52.948478 17643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:52.948537 17643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:52.948583 17643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:53:52.950214 17643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:52.950393 17643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:52.950564 17643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:52.955755 17649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:52.955823 17643 server_base.cc:1034] running on GCE node
W20260928 22:53:52.955755 17651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:52.955797 17648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:52.956148 17643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:52.956342 17643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:52.957487 17643 hybrid_clock.cc:648] HybridClock initialized: now 1790636032957461 us; error 41 us; skew 500 ppm
Sep 28 22:53:52 dist-test-slave-kzfr krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:52.960105 17643 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:53:52.960589 17643 webserver.cc:460] Webserver started at http://127.14.14.130:44027/ using document root <none> and password file <none>
I20260928 22:53:52.960850 17643 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:52.960896 17643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:52.961043 17643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:52.962153 17643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "c297e5d946f04298a59e5e876e16219b"
format_stamp: "Formatted at 2026-09-28 22:53:52 on dist-test-slave-kzfr"
server_key: "02c2e3757368002f03fa6f3fa4d05872"
server_key_iv: "cf2bf344749f9909a19773f9dcb26386"
server_key_version: "encryptionkey@0"
I20260928 22:53:52.962507 17643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "c297e5d946f04298a59e5e876e16219b"
format_stamp: "Formatted at 2026-09-28 22:53:52 on dist-test-slave-kzfr"
server_key: "02c2e3757368002f03fa6f3fa4d05872"
server_key_iv: "cf2bf344749f9909a19773f9dcb26386"
server_key_version: "encryptionkey@0"
I20260928 22:53:52.963840 17643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:53:52.964578 17658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:52.964819 17643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:52.964884 17643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "c297e5d946f04298a59e5e876e16219b"
format_stamp: "Formatted at 2026-09-28 22:53:52 on dist-test-slave-kzfr"
server_key: "02c2e3757368002f03fa6f3fa4d05872"
server_key_iv: "cf2bf344749f9909a19773f9dcb26386"
server_key_version: "encryptionkey@0"
I20260928 22:53:52.964975 17643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:52.974193 17643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:52.976415 17643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:52.976570 17643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:52.976804 17643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:52.977105 17643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:52.977140 17643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:52.977162 17643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:52.977209 17643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:52.983167 17643 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:42359
I20260928 22:53:52.983219 17771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:42359 every 8 connection(s)
I20260928 22:53:52.983512 17643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:53:52 dist-test-slave-kzfr krb5kdc[17402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:52.989792 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 17643
I20260928 22:53:52.989887 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:53:52.990095 14394 external_mini_cluster.cc:1428] Setting key 28e8c95f59422a0529d045158efa7258
I20260928 22:53:52.993162 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:52.984239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:33593 (local address 127.14.14.190:46023)
0928 22:53:52.984285 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:52.984286 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:52.984495 (+ 209us) server_negotiation.cc:407] Connection header received
0928 22:53:52.984986 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:52.984987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:52.984998 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:52.985057 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:52.986035 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:52.986554 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:52.987579 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:52.987790 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:52.987851 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:52.990286 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:52.990288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:52.990288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:52.990295 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:52.991741 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:52.992175 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:52.992177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:52.992177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:52.992238 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:52.992502 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:52.992503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:52.992503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:52.992625 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:52.992701 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:52.992988 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 22:53:52.993072 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:53:52.993948 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:52.984228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:46023 (local address 127.14.14.130:33593)
0928 22:53:52.984375 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:52.984408 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:52.984875 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:52.985110 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:52.985113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:52.985742 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:52.985941 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:52.985946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:52.986673 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:52.986674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:52.987497 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:52.987499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:52.987870 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:52.987871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:52.987972 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:52.988505 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:52.988513 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:52.990169 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:52.991844 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:52.991852 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:52.991854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:52.992081 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:52.992313 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:52.992314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:52.992314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:52.992427 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:52.992693 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:52.992700 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:52.992846 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 22:53:52.992953 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 22:53:52.993127 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
I20260928 22:53:52.994464 17772 heartbeater.cc:347] Connected to a master server at 127.14.14.190:46023
I20260928 22:53:52.994575 17772 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:52.994764 17772 heartbeater.cc:511] Master 127.14.14.190:46023 requested a full tablet report, sending...
I20260928 22:53:52.995209 17447 ts_manager.cc:194] Registered new tserver with Master: c297e5d946f04298a59e5e876e16219b (127.14.14.130:42359)
I20260928 22:53:52.995540 17447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:33593
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:53:53.017031 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46023
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:53.097939 17777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:53.098160 17777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:53.098196 17777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:53.099325 17777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:53.099390 17777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:53.099436 17777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:53:53.100910 17777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:53.101174 17777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:53.101403 17777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:53.106599 17783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:53.106593 17782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:53.106709 17777 server_base.cc:1034] running on GCE node
W20260928 22:53:53.106704 17785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:53.107561 17777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:53.107832 17777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:53.109005 17777 hybrid_clock.cc:648] HybridClock initialized: now 1790636033108991 us; error 51 us; skew 500 ppm
Sep 28 22:53:53 dist-test-slave-kzfr krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636033, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:53.111560 17777 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:53:53.111987 17777 webserver.cc:460] Webserver started at http://127.14.14.131:37129/ using document root <none> and password file <none>
I20260928 22:53:53.112224 17777 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:53.112294 17777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:53.112454 17777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:53.113492 17777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "1f876aa77a424c43a7f3b9ee596c32b7"
format_stamp: "Formatted at 2026-09-28 22:53:53 on dist-test-slave-kzfr"
server_key: "4a9292f64241f8e9574a9064acd901dd"
server_key_iv: "1a795b5d9ae34bd8653be3cb8d7c1eef"
server_key_version: "encryptionkey@0"
I20260928 22:53:53.113920 17777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "1f876aa77a424c43a7f3b9ee596c32b7"
format_stamp: "Formatted at 2026-09-28 22:53:53 on dist-test-slave-kzfr"
server_key: "4a9292f64241f8e9574a9064acd901dd"
server_key_iv: "1a795b5d9ae34bd8653be3cb8d7c1eef"
server_key_version: "encryptionkey@0"
I20260928 22:53:53.115247 17777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:53.115998 17792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:53.116160 17777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:53.116268 17777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "1f876aa77a424c43a7f3b9ee596c32b7"
format_stamp: "Formatted at 2026-09-28 22:53:53 on dist-test-slave-kzfr"
server_key: "4a9292f64241f8e9574a9064acd901dd"
server_key_iv: "1a795b5d9ae34bd8653be3cb8d7c1eef"
server_key_version: "encryptionkey@0"
I20260928 22:53:53.116369 17777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:53.126591 17777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:53.128252 17777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:53.128409 17777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:53.128619 17777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:53.128911 17777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:53.128966 17777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:53.129006 17777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:53.129035 17777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:53.134830 17777 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:37569
I20260928 22:53:53.134907 17905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:37569 every 8 connection(s)
I20260928 22:53:53.135205 17777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:53:53 dist-test-slave-kzfr krb5kdc[17402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636033, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:53.142022 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 17777
I20260928 22:53:53.142128 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:53:53.142326 14394 external_mini_cluster.cc:1428] Setting key 60b8b8dc686bd2c37d60ba4e86f32bf7
I20260928 22:53:53.144275 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:53.135894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:50189 (local address 127.14.14.190:46023)
0928 22:53:53.135981 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:53.135984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:53.136135 (+ 151us) server_negotiation.cc:407] Connection header received
0928 22:53:53.136580 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:53.136581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:53.136591 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:53.136642 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:53.137489 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:53.138043 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:53.138995 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:53.139214 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:53.139270 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:53.141275 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:53.141276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:53.141277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:53.141283 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:53.143036 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:53.143441 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:53.143441 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:53.143442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:53.143492 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:53.143699 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:53.143699 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:53.143699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:53.143797 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:53.143847 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:53.144146 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 22:53:53.144215 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 22:53:53.144439 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:53.135845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:46023 (local address 127.14.14.131:50189)
0928 22:53:53.136016 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:53.136050 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:53.136498 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:53.136691 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:53.136694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:53.137233 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:53.137416 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:53.137421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:53.138115 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:53.138116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:53.138910 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:53.138912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:53.139307 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:53.139308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:53.139403 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:53.139825 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:53.139833 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:53.141189 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:53.143117 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:53.143124 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:53.143125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:53.143377 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:53.143552 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:53.143552 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:53.143553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:53.143646 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:53.143851 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:53.143856 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:53.144033 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 22:53:53.144119 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 22:53:53.144232 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260928 22:53:53.144884 17906 heartbeater.cc:347] Connected to a master server at 127.14.14.190:46023
I20260928 22:53:53.144995 17906 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:53.145165 17906 heartbeater.cc:511] Master 127.14.14.190:46023 requested a full tablet report, sending...
I20260928 22:53:53.145517 17447 ts_manager.cc:194] Registered new tserver with Master: 1f876aa77a424c43a7f3b9ee596c32b7 (127.14.14.131:37569)
I20260928 22:53:53.145893 17447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:50189
I20260928 22:53:53.153867 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 22:53:53 dist-test-slave-kzfr krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636033, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 22:53:53 dist-test-slave-kzfr krb5kdc[17402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636033, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:53.176126 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:53.169076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.14.14.190:46023)
0928 22:53:53.169138 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:53.169140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:53.169308 (+ 168us) server_negotiation.cc:407] Connection header received
0928 22:53:53.169336 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:53.169337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:53.169347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:53.169393 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:53.170141 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:53.170610 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:53.171349 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:53.171531 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:53.171590 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:53.173793 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:53.173795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:53.173795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:53.173803 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:53.174964 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:53.175313 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:53.175314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:53.175315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:53.175374 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:53.175579 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:53.175580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:53.175580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:53.175729 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:53.175838 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:53.175947 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 22:53:53.176025 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 22:53:53.176342 17447 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49726
I20260928 22:53:53.176820 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 17509
I20260928 22:53:53.181384 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 17643
I20260928 22:53:53.185400 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 17777
I20260928 22:53:53.189733 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 17415
2026-09-28T22:53:53Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1821 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:53 dist-test-slave-kzfr krb5kdc[17918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:53 dist-test-slave-kzfr krb5kdc[17918](info): set up 2 sockets
Sep 28 22:53:53 dist-test-slave-kzfr krb5kdc[17918](info): commencing operation
krb5kdc: starting...
W20260928 22:53:54.233999 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:53:54 dist-test-slave-kzfr krb5kdc[17918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:53:54.356889 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44927
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:44927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:54.437263 17931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:54.437511 17931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:54.437549 17931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:54.438802 17931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:54.438863 17931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:54.438886 17931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:54.438915 17931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:54.438938 17931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:54.440495 17931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:44927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:54.440800 17931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:54.441035 17931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:54.446077 17936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:54.446154 17931 server_base.cc:1034] running on GCE node
W20260928 22:53:54.446121 17937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:54.446121 17939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:54.446532 17931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:54.446830 17931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:54.448014 17931 hybrid_clock.cc:648] HybridClock initialized: now 1790636034447973 us; error 61 us; skew 500 ppm
Sep 28 22:53:54 dist-test-slave-kzfr krb5kdc[17918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636034, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:54.450575 17931 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:53:54.451123 17931 webserver.cc:460] Webserver started at http://127.14.14.190:34517/ using document root <none> and password file <none>
I20260928 22:53:54.451390 17931 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:54.451432 17931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:54.451520 17931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:54.452515 17931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "225b3d22fc944a7ca22729ff4768c8f5"
format_stamp: "Formatted at 2026-09-28 22:53:54 on dist-test-slave-kzfr"
server_key: "46bdcd053bea4ed254facbfa52fad068"
server_key_iv: "f25b54ae905fe01548fe6c8cb9178434"
server_key_version: "encryptionkey@0"
I20260928 22:53:54.452813 17931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "225b3d22fc944a7ca22729ff4768c8f5"
format_stamp: "Formatted at 2026-09-28 22:53:54 on dist-test-slave-kzfr"
server_key: "46bdcd053bea4ed254facbfa52fad068"
server_key_iv: "f25b54ae905fe01548fe6c8cb9178434"
server_key_version: "encryptionkey@0"
I20260928 22:53:54.454128 17931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 22:53:54.454870 17946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:54.455051 17931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:54.455153 17931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "225b3d22fc944a7ca22729ff4768c8f5"
format_stamp: "Formatted at 2026-09-28 22:53:54 on dist-test-slave-kzfr"
server_key: "46bdcd053bea4ed254facbfa52fad068"
server_key_iv: "f25b54ae905fe01548fe6c8cb9178434"
server_key_version: "encryptionkey@0"
I20260928 22:53:54.455237 17931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:54.479085 17931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:54.481184 17931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:54.481360 17931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:54.485251 17931 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:44927
I20260928 22:53:54.485311 17998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:44927 every 8 connection(s)
I20260928 22:53:54.485862 17931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:54.486580 17999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:54.488920 17999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5: Bootstrap starting.
I20260928 22:53:54.489522 17999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:54.489877 17999 log.cc:824] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:54.490550 17999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5: No bootstrap required, opened a new log
I20260928 22:53:54.491818 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 17931
I20260928 22:53:54.491955 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:53:54.492156 14394 external_mini_cluster.cc:1428] Setting key 6c97e72f11c064f87ed0e1d078d0fa42
I20260928 22:53:54.492292 17999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225b3d22fc944a7ca22729ff4768c8f5" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44927 } }
I20260928 22:53:54.492499 17999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:54.492545 17999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 225b3d22fc944a7ca22729ff4768c8f5, State: Initialized, Role: FOLLOWER
I20260928 22:53:54.492738 17999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225b3d22fc944a7ca22729ff4768c8f5" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44927 } }
I20260928 22:53:54.492834 17999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:54.492869 17999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:54.492933 17999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:54.493571 17999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225b3d22fc944a7ca22729ff4768c8f5" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44927 } }
I20260928 22:53:54.493717 17999 leader_election.cc:302] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 225b3d22fc944a7ca22729ff4768c8f5; no voters:
I20260928 22:53:54.493878 17999 leader_election.cc:288] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:54.493906 18004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:54.494011 18004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [term 1 LEADER]: Becoming Leader. State: Replica: 225b3d22fc944a7ca22729ff4768c8f5, State: Running, Role: LEADER
I20260928 22:53:54.494091 18004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225b3d22fc944a7ca22729ff4768c8f5" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44927 } }
I20260928 22:53:54.494138 17999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:54.494405 18004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "225b3d22fc944a7ca22729ff4768c8f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225b3d22fc944a7ca22729ff4768c8f5" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44927 } } }
I20260928 22:53:54.494555 18004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:54.494745 18004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 225b3d22fc944a7ca22729ff4768c8f5. Latest consensus state: current_term: 1 leader_uuid: "225b3d22fc944a7ca22729ff4768c8f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225b3d22fc944a7ca22729ff4768c8f5" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44927 } } }
I20260928 22:53:54.494870 18004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:54.495234 18011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:54.495801 18011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:54.497464 18011 catalog_manager.cc:1347] Generated new cluster ID: abf9ee17de7247058b89da0978fff7b2
I20260928 22:53:54.497532 18011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:53:54 dist-test-slave-kzfr krb5kdc[17918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:54.504268 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:54.492716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34376 (local address 127.14.14.190:44927)
0928 22:53:54.493333 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:54.493343 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:54.493379 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:53:54.493493 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:54.493501 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:54.493559 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:54.493734 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:54.494624 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:54.496078 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:54.497123 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:54.497745 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:54.497858 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:54.500621 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:54.500626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:54.500626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:54.500638 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:54.502539 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:54.502934 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:54.502937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:54.502937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:54.503043 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:54.503264 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:54.503265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:54.503265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:54.503557 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:54.503694 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:54.503827 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 22:53:54.504049 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":504,"threads_started":1}
I20260928 22:53:54.507259 18011 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:54.507963 18011 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:54.515167 18011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 225b3d22fc944a7ca22729ff4768c8f5: Generated new TSK 0
I20260928 22:53:54.515360 18011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:53:54.553174 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44927
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:54.652359 18025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:54.652591 18025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:54.652640 18025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:54.653887 18025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:54.653949 18025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:54.653996 18025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:53:54.655500 18025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:54.655772 18025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:54.655994 18025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:54.661183 18031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:54.661195 18033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:54.661394 18025 server_base.cc:1034] running on GCE node
W20260928 22:53:54.661195 18030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:54.661734 18025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:54.661980 18025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:54.663137 18025 hybrid_clock.cc:648] HybridClock initialized: now 1790636034663121 us; error 37 us; skew 500 ppm
Sep 28 22:53:54 dist-test-slave-kzfr krb5kdc[17918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636034, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:54.665866 18025 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:53:54.666280 18025 webserver.cc:460] Webserver started at http://127.14.14.129:45973/ using document root <none> and password file <none>
I20260928 22:53:54.666522 18025 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:54.666592 18025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:54.666747 18025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:54.667783 18025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "5d84e68bf50b4d0cafb56bbd0958134d"
format_stamp: "Formatted at 2026-09-28 22:53:54 on dist-test-slave-kzfr"
server_key: "2103eb6403ed3574aadc4e17e522b987"
server_key_iv: "19a02a430784bdd40c2e37f09c11ff27"
server_key_version: "encryptionkey@0"
I20260928 22:53:54.668136 18025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "5d84e68bf50b4d0cafb56bbd0958134d"
format_stamp: "Formatted at 2026-09-28 22:53:54 on dist-test-slave-kzfr"
server_key: "2103eb6403ed3574aadc4e17e522b987"
server_key_iv: "19a02a430784bdd40c2e37f09c11ff27"
server_key_version: "encryptionkey@0"
I20260928 22:53:54.669441 18025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:53:54.670231 18040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:54.670409 18025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:53:54.670506 18025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "5d84e68bf50b4d0cafb56bbd0958134d"
format_stamp: "Formatted at 2026-09-28 22:53:54 on dist-test-slave-kzfr"
server_key: "2103eb6403ed3574aadc4e17e522b987"
server_key_iv: "19a02a430784bdd40c2e37f09c11ff27"
server_key_version: "encryptionkey@0"
I20260928 22:53:54.670593 18025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:54.687081 18025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:54.689021 18025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:54.689188 18025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:54.689427 18025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:54.689785 18025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:54.689822 18025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:54.689875 18025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:54.689903 18025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:54.695752 18025 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:44335
I20260928 22:53:54.695873 18153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:44335 every 8 connection(s)
I20260928 22:53:54.696156 18025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:53:54.699005 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 18025
I20260928 22:53:54.699095 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:53:54.699331 14394 external_mini_cluster.cc:1428] Setting key 0b29c14e29c71f5e80f6643dcf0893ad
Sep 28 22:53:54 dist-test-slave-kzfr krb5kdc[17918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636034, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:54.706594 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:54.696969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:58229 (local address 127.14.14.190:44927)
0928 22:53:54.697012 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:54.697014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:54.697262 (+ 248us) server_negotiation.cc:407] Connection header received
0928 22:53:54.698291 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:54.698292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:54.698316 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:54.698370 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:54.699006 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:54.699716 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:54.700799 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:54.701028 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:54.701114 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:54.703619 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:54.703621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:54.703621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:54.703630 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:54.705064 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:54.705566 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:54.705567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:54.705567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:54.705671 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:54.705968 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:54.705968 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:54.705969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:54.706071 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:54.706123 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:54.706468 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 22:53:54.706532 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:53:54.706946 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:54.696956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44927 (local address 127.14.14.129:58229)
0928 22:53:54.697139 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:54.697175 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:54.698175 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:54.698449 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:54.698453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:54.698703 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:54.698896 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:54.698899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:54.699835 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:54.699836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:54.700707 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:54.700709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:54.701126 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:54.701127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:54.701254 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:54.701832 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:54.701840 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:54.703522 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:54.705161 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:54.705168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:54.705170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:54.705489 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:54.705744 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:54.705745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:54.705746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:54.705896 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:54.706150 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:54.706157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:54.706327 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 22:53:54.706450 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 22:53:54.706620 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":99,"threads_started":1}
I20260928 22:53:54.707535 18154 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44927
I20260928 22:53:54.707636 18154 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
I20260928 22:53:54.707847 18154 heartbeater.cc:511] Master 127.14.14.190:44927 requested a full tablet report, sending...
I20260928 22:53:54.708325 17963 ts_manager.cc:194] Registered new tserver with Master: 5d84e68bf50b4d0cafb56bbd0958134d (127.14.14.129:44335)
I20260928 22:53:54.709041 17963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:58229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:53:54.726574 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44927
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:54.808305 18159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:54.808555 18159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:54.808640 18159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:54.809839 18159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:54.809908 18159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:54.809957 18159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:53:54.811484 18159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:54.811760 18159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:54.812000 18159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:54.817147 18164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:54.817232 18165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:54.817122 18167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:54.817349 18159 server_base.cc:1034] running on GCE node
I20260928 22:53:54.817762 18159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:54.818044 18159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:54.819203 18159 hybrid_clock.cc:648] HybridClock initialized: now 1790636034819191 us; error 34 us; skew 500 ppm
Sep 28 22:53:54 dist-test-slave-kzfr krb5kdc[17918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636034, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:54.821867 18159 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:53:54.822358 18159 webserver.cc:460] Webserver started at http://127.14.14.130:34915/ using document root <none> and password file <none>
I20260928 22:53:54.822542 18159 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:54.822582 18159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:54.822682 18159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:54.823693 18159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "d52ee2dfe47946e085415de42cca732e"
format_stamp: "Formatted at 2026-09-28 22:53:54 on dist-test-slave-kzfr"
server_key: "e87aa241bb48ef5e9d87a0c899991589"
server_key_iv: "d93681cc016a153d318643675cc6f92a"
server_key_version: "encryptionkey@0"
I20260928 22:53:54.823992 18159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "d52ee2dfe47946e085415de42cca732e"
format_stamp: "Formatted at 2026-09-28 22:53:54 on dist-test-slave-kzfr"
server_key: "e87aa241bb48ef5e9d87a0c899991589"
server_key_iv: "d93681cc016a153d318643675cc6f92a"
server_key_version: "encryptionkey@0"
I20260928 22:53:54.825260 18159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:54.826083 18174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:54.826249 18159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:54.826359 18159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "d52ee2dfe47946e085415de42cca732e"
format_stamp: "Formatted at 2026-09-28 22:53:54 on dist-test-slave-kzfr"
server_key: "e87aa241bb48ef5e9d87a0c899991589"
server_key_iv: "d93681cc016a153d318643675cc6f92a"
server_key_version: "encryptionkey@0"
I20260928 22:53:54.826436 18159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:54.842582 18159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:54.844411 18159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:54.844584 18159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:54.844831 18159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:54.845151 18159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:54.845208 18159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:54.845249 18159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:54.845278 18159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:54.851052 18159 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:37481
I20260928 22:53:54.851104 18287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:37481 every 8 connection(s)
I20260928 22:53:54.851413 18159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:53:54 dist-test-slave-kzfr krb5kdc[17918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636034, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:54.860877 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:54.852158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:47775 (local address 127.14.14.190:44927)
0928 22:53:54.852205 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:54.852207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:54.852408 (+ 201us) server_negotiation.cc:407] Connection header received
0928 22:53:54.853336 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:54.853338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:54.853350 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:54.853411 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:54.854086 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:54.854609 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:54.855595 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:54.855770 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:54.855819 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:54.858108 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:54.858109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:54.858110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:54.858116 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:54.859503 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:54.860002 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:54.860003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:54.860004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:54.860052 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:54.860325 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:54.860326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:54.860326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:54.860420 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:54.860470 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:54.860752 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 22:53:54.860815 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:53:54.861040 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:54.852113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44927 (local address 127.14.14.130:47775)
0928 22:53:54.852274 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:54.852315 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:54.853223 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:54.853488 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:54.853492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:54.853822 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:54.854014 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:54.854018 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:54.854710 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:54.854711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:54.855523 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:54.855525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:54.855840 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:54.855841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:54.855940 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:54.856412 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:54.856420 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:54.858027 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:54.859604 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:54.859610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:54.859612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:54.859929 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:54.860134 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:54.860135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:54.860135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:54.860247 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:54.860502 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:54.860508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:54.860640 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 22:53:54.860729 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 22:53:54.860844 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260928 22:53:54.861491 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 18159
I20260928 22:53:54.861599 18288 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44927
I20260928 22:53:54.861632 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:53:54.861718 18288 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:54.861852 14394 external_mini_cluster.cc:1428] Setting key c250886b9162c574b7ad8ae2b3b33fa3
I20260928 22:53:54.861907 18288 heartbeater.cc:511] Master 127.14.14.190:44927 requested a full tablet report, sending...
I20260928 22:53:54.862304 17962 ts_manager.cc:194] Registered new tserver with Master: d52ee2dfe47946e085415de42cca732e (127.14.14.130:37481)
I20260928 22:53:54.862703 17962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:47775
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:53:54.887979 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44927
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:54.970300 18293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:54.970505 18293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:54.970530 18293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:54.971753 18293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:54.971796 18293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:54.971822 18293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:53:54.973629 18293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:54.973836 18293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:54.974021 18293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:54.979090 18299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:54.979264 18293 server_base.cc:1034] running on GCE node
W20260928 22:53:54.979143 18298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:54.979105 18301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:54.979588 18293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:54.979928 18293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:54.981123 18293 hybrid_clock.cc:648] HybridClock initialized: now 1790636034981101 us; error 49 us; skew 500 ppm
Sep 28 22:53:54 dist-test-slave-kzfr krb5kdc[17918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636034, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:54.983824 18293 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:53:54.984316 18293 webserver.cc:460] Webserver started at http://127.14.14.131:45683/ using document root <none> and password file <none>
I20260928 22:53:54.984557 18293 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:54.984627 18293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:54.984786 18293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:54.985896 18293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "89fe6a7740b14804b81b6d12f780fbef"
format_stamp: "Formatted at 2026-09-28 22:53:54 on dist-test-slave-kzfr"
server_key: "6276aa01e1aeb3b967ef1597fffec744"
server_key_iv: "45663dcd690000b2fe2a31d934927845"
server_key_version: "encryptionkey@0"
I20260928 22:53:54.986249 18293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "89fe6a7740b14804b81b6d12f780fbef"
format_stamp: "Formatted at 2026-09-28 22:53:54 on dist-test-slave-kzfr"
server_key: "6276aa01e1aeb3b967ef1597fffec744"
server_key_iv: "45663dcd690000b2fe2a31d934927845"
server_key_version: "encryptionkey@0"
I20260928 22:53:54.987560 18293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:53:54.988631 18308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:54.988900 18293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:54.989039 18293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "89fe6a7740b14804b81b6d12f780fbef"
format_stamp: "Formatted at 2026-09-28 22:53:54 on dist-test-slave-kzfr"
server_key: "6276aa01e1aeb3b967ef1597fffec744"
server_key_iv: "45663dcd690000b2fe2a31d934927845"
server_key_version: "encryptionkey@0"
I20260928 22:53:54.989126 18293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:55.011792 18293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:55.013973 18293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:55.014146 18293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:55.014394 18293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:55.014729 18293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:55.014784 18293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:55.014827 18293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:55.014855 18293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:55.020752 18293 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:36647
I20260928 22:53:55.020797 18421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:36647 every 8 connection(s)
I20260928 22:53:55.021122 18293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:53:55.022971 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 18293
I20260928 22:53:55.023093 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:53:55.023286 14394 external_mini_cluster.cc:1428] Setting key 485c802bcb8499934dc53fbdd5d4ed6e
Sep 28 22:53:55 dist-test-slave-kzfr krb5kdc[17918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636034, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:55.030673 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:55.021925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:58239 (local address 127.14.14.190:44927)
0928 22:53:55.021975 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:55.021977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:55.022154 (+ 177us) server_negotiation.cc:407] Connection header received
0928 22:53:55.023096 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:55.023098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:55.023112 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:55.023197 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:55.023871 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:55.024375 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:55.025393 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:55.025666 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:55.025724 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:55.027926 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:55.027928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:55.027928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:55.027935 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:55.029168 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:55.029675 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:55.029676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:55.029677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:55.029734 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:55.030000 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:55.030001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:55.030001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:55.030122 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:55.030225 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:55.030497 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 22:53:55.030582 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:53:55.030872 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:55.021878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44927 (local address 127.14.14.131:58239)
0928 22:53:55.022039 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:55.022082 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:55.022971 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:55.023251 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:55.023255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:55.023567 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:55.023768 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:55.023773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:55.024474 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:55.024475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:55.025300 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:55.025302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:55.025742 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:55.025744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:55.025865 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:55.026291 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:55.026298 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:55.027827 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:55.029278 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:55.029285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:55.029287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:55.029539 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:55.029809 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:55.029810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:55.029811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:55.029926 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:55.030214 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:55.030220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:55.030363 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 22:53:55.030473 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 22:53:55.030624 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260928 22:53:55.031368 18422 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44927
I20260928 22:53:55.031464 18422 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:55.031601 18422 heartbeater.cc:511] Master 127.14.14.190:44927 requested a full tablet report, sending...
I20260928 22:53:55.031988 17962 ts_manager.cc:194] Registered new tserver with Master: 89fe6a7740b14804b81b6d12f780fbef (127.14.14.131:36647)
I20260928 22:53:55.032353 17962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:58239
I20260928 22:53:55.034803 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 22:53:55 dist-test-slave-kzfr krb5kdc[17918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 22:53:55 dist-test-slave-kzfr krb5kdc[17918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.129@KRBTEST.COM
I20260928 22:53:55.057526 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:55.049671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.14.14.129:44335)
0928 22:53:55.049849 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:55.049855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:55.049870 (+ 15us) server_negotiation.cc:407] Connection header received
0928 22:53:55.049903 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:55.049910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:55.049926 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:55.050015 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:55.050604 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:55.051345 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:55.052155 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:55.052485 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:55.052569 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:55.054816 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:55.054832 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:55.054832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:55.054847 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:55.056250 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:55.056628 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:55.056633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:55.056634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:55.056705 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:55.056923 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:55.056924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:55.056925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:55.057066 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:55.057223 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:55.057308 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 22:53:55.057394 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
Sep 28 22:53:55 dist-test-slave-kzfr krb5kdc[17918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:55.065052 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:55.058807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34390 (local address 127.14.14.190:44927)
0928 22:53:55.058871 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:55.058873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:55.058904 (+ 31us) server_negotiation.cc:407] Connection header received
0928 22:53:55.058957 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:55.058958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:55.058967 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:55.059015 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:55.059445 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:55.059910 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:55.060561 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:55.060749 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:55.060797 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:55.062959 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:55.062961 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:55.062961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:55.062969 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:55.064108 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:55.064404 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:55.064405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:55.064405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:55.064454 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:55.064613 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:55.064614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:55.064614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:55.064746 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:55.064818 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:55.064929 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 22:53:55.064986 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 22:53:55.065301 17962 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34390
I20260928 22:53:55.065749 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 18025
I20260928 22:53:55.070349 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 18159
I20260928 22:53:55.075055 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 18293
I20260928 22:53:55.079701 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 17931
2026-09-28T22:53:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1890 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:55 dist-test-slave-kzfr krb5kdc[18434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:55 dist-test-slave-kzfr krb5kdc[18434](info): set up 2 sockets
Sep 28 22:53:55 dist-test-slave-kzfr krb5kdc[18434](info): commencing operation
krb5kdc: starting...
W20260928 22:53:56.133891 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:53:56 dist-test-slave-kzfr krb5kdc[18434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:53:56.253501 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:39673
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:37429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:39673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:56.333941 18447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:56.334138 18447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:56.334161 18447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:56.335263 18447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:56.335300 18447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:56.335314 18447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:56.335327 18447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:56.335340 18447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:56.336743 18447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:37429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:39673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:39673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:56.336925 18447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:56.337080 18447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:56.341776 18453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:56.341806 18452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:56.341883 18447 server_base.cc:1034] running on GCE node
W20260928 22:53:56.341781 18455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:56.342222 18447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:56.342528 18447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:56.343686 18447 hybrid_clock.cc:648] HybridClock initialized: now 1790636036343673 us; error 49 us; skew 500 ppm
Sep 28 22:53:56 dist-test-slave-kzfr krb5kdc[18434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:56.346115 18447 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:53:56.346460 18447 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:56.346534 18447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:56.346686 18447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:56.347749 18447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "b8471e661291436f9e726369246fa280"
format_stamp: "Formatted at 2026-09-28 22:53:56 on dist-test-slave-kzfr"
server_key: "ca49aa5345557758fd2db677ccd15420"
server_key_iv: "7b8940ba53cd2258837d832d71b555aa"
server_key_version: "encryptionkey@0"
I20260928 22:53:56.348100 18447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "b8471e661291436f9e726369246fa280"
format_stamp: "Formatted at 2026-09-28 22:53:56 on dist-test-slave-kzfr"
server_key: "ca49aa5345557758fd2db677ccd15420"
server_key_iv: "7b8940ba53cd2258837d832d71b555aa"
server_key_version: "encryptionkey@0"
I20260928 22:53:56.349359 18447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:53:56.350088 18461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:56.350247 18447 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:56.350353 18447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "b8471e661291436f9e726369246fa280"
format_stamp: "Formatted at 2026-09-28 22:53:56 on dist-test-slave-kzfr"
server_key: "ca49aa5345557758fd2db677ccd15420"
server_key_iv: "7b8940ba53cd2258837d832d71b555aa"
server_key_version: "encryptionkey@0"
I20260928 22:53:56.350435 18447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:56.358726 18447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:56.360929 18447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:56.361084 18447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:56.364827 18447 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:39673
I20260928 22:53:56.364854 18513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:39673 every 8 connection(s)
I20260928 22:53:56.365124 18447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:56.366107 18514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:56.367985 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 18447
I20260928 22:53:56.368114 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:53:56.368280 14394 external_mini_cluster.cc:1428] Setting key e06380796f7f5d72d7079c5de6fb7e0a
I20260928 22:53:56.368384 18514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280: Bootstrap starting.
I20260928 22:53:56.369164 18514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:56.369500 18514 log.cc:824] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:56.370263 18514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280: No bootstrap required, opened a new log
I20260928 22:53:56.372565 18514 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8471e661291436f9e726369246fa280" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39673 } }
I20260928 22:53:56.372727 18514 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:56.372766 18514 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8471e661291436f9e726369246fa280, State: Initialized, Role: FOLLOWER
I20260928 22:53:56.372889 18514 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8471e661291436f9e726369246fa280" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39673 } }
I20260928 22:53:56.372959 18514 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:56.372987 18514 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:56.373031 18514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:56.373755 18514 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8471e661291436f9e726369246fa280" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39673 } }
I20260928 22:53:56.373895 18514 leader_election.cc:302] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8471e661291436f9e726369246fa280; no voters:
I20260928 22:53:56.374066 18514 leader_election.cc:288] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:56.374128 18519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:56.374286 18519 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [term 1 LEADER]: Becoming Leader. State: Replica: b8471e661291436f9e726369246fa280, State: Running, Role: LEADER
I20260928 22:53:56.374336 18514 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:56.374475 18519 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8471e661291436f9e726369246fa280" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39673 } }
I20260928 22:53:56.374902 18521 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8471e661291436f9e726369246fa280. Latest consensus state: current_term: 1 leader_uuid: "b8471e661291436f9e726369246fa280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8471e661291436f9e726369246fa280" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39673 } } }
I20260928 22:53:56.374985 18521 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:56.375216 18520 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8471e661291436f9e726369246fa280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8471e661291436f9e726369246fa280" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39673 } } }
I20260928 22:53:56.375339 18520 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:56.375501 18526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 22:53:56 dist-test-slave-kzfr krb5kdc[18434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:56.376085 18526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:56.377981 18526 catalog_manager.cc:1347] Generated new cluster ID: a1da45515049493092e7433c7126a7f6
I20260928 22:53:56.378053 18526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:56.380285 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.368857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48832 (local address 127.14.14.190:39673)
0928 22:53:56.369042 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:56.369051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:56.369085 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:53:56.369719 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:56.369727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:56.369788 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:56.369910 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:56.370576 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.371923 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.372734 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.373337 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.373460 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.376451 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:56.376455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:56.376456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:56.376467 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:56.378666 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:56.379059 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:56.379062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:56.379063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:56.379175 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:56.379380 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:56.379381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:56.379382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:56.379696 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:56.379852 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:56.379931 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 22:53:56.380071 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260928 22:53:56.386976 18526 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:56.387756 18526 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:56.390179 18526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280: Generated new TSK 0
I20260928 22:53:56.390364 18526 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:53:56.420600 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:39673
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:37429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:56.503028 18540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:56.503257 18540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:56.503285 18540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:56.504419 18540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:56.504456 18540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:56.504482 18540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:53:56.506050 18540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:37429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:39673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:56.506237 18540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:56.506419 18540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:53:56.511718 18540 server_base.cc:1034] running on GCE node
W20260928 22:53:56.511664 18546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:56.511793 18545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:56.511652 18548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:56.512118 18540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:56.512413 18540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:56.513665 18540 hybrid_clock.cc:648] HybridClock initialized: now 1790636036513647 us; error 30 us; skew 500 ppm
Sep 28 22:53:56 dist-test-slave-kzfr krb5kdc[18434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:56.516336 18540 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:53:56.516709 18540 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:56.516757 18540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:56.516852 18540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:56.517946 18540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "b6b4a3b3f4c24d17a0c6e0658448552b"
format_stamp: "Formatted at 2026-09-28 22:53:56 on dist-test-slave-kzfr"
server_key: "ab35f5ce73f4ef1869783e40bbb3a03e"
server_key_iv: "766f3533316001bbacf5ea6e3952091c"
server_key_version: "encryptionkey@0"
I20260928 22:53:56.518247 18540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "b6b4a3b3f4c24d17a0c6e0658448552b"
format_stamp: "Formatted at 2026-09-28 22:53:56 on dist-test-slave-kzfr"
server_key: "ab35f5ce73f4ef1869783e40bbb3a03e"
server_key_iv: "766f3533316001bbacf5ea6e3952091c"
server_key_version: "encryptionkey@0"
I20260928 22:53:56.519528 18540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:53:56.520221 18554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:56.520424 18540 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:56.520516 18540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "b6b4a3b3f4c24d17a0c6e0658448552b"
format_stamp: "Formatted at 2026-09-28 22:53:56 on dist-test-slave-kzfr"
server_key: "ab35f5ce73f4ef1869783e40bbb3a03e"
server_key_iv: "766f3533316001bbacf5ea6e3952091c"
server_key_version: "encryptionkey@0"
I20260928 22:53:56.520572 18540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:56.539676 18540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:56.541450 18540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:56.541558 18540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:56.541812 18540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:56.542079 18540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:56.542110 18540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:56.542132 18540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:56.542150 18540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:56.547962 18540 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:41789
I20260928 22:53:56.548043 18667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:41789 every 8 connection(s)
I20260928 22:53:56.548312 18540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:53:56 dist-test-slave-kzfr krb5kdc[18434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:56.555521 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 18540
I20260928 22:53:56.555640 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:53:56.555874 14394 external_mini_cluster.cc:1428] Setting key 811fdfe459dec5324352146a91998a14
I20260928 22:53:56.557895 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.549043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:36647 (local address 127.14.14.190:39673)
0928 22:53:56.549116 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:56.549118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:56.549320 (+ 202us) server_negotiation.cc:407] Connection header received
0928 22:53:56.549930 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:56.549932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:56.549943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:56.550001 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:56.550945 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.551470 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.552482 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.552706 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.552798 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.554951 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:56.554953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:56.554953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:56.554960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:56.556377 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:56.556859 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:56.556861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:56.556861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:56.556926 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:56.557158 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:56.557159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:56.557160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:56.557309 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:56.557379 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:56.557641 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 22:53:56.557743 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 22:53:56.558024 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.549036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:39673 (local address 127.14.14.129:36647)
0928 22:53:56.549199 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:56.549233 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:56.549796 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:56.550075 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:56.550079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:56.550331 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:56.550816 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.550821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.551564 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.551565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.552389 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.552391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.552827 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.552828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.552963 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.553501 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:56.553509 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:56.554861 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:56.556514 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:56.556521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:56.556523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:56.556770 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:56.556998 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:56.556999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:56.556999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:56.557094 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:56.557379 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:56.557383 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:56.557496 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 22:53:56.557630 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 22:53:56.557766 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260928 22:53:56.558781 18668 heartbeater.cc:347] Connected to a master server at 127.14.14.190:39673
I20260928 22:53:56.558889 18668 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:56.559060 18668 heartbeater.cc:511] Master 127.14.14.190:39673 requested a full tablet report, sending...
I20260928 22:53:56.559583 18478 ts_manager.cc:194] Registered new tserver with Master: b6b4a3b3f4c24d17a0c6e0658448552b (127.14.14.129:41789)
I20260928 22:53:56.560290 18478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:36647
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:53:56.580487 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:39673
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:37429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:56.659868 18673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:56.660095 18673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:56.660136 18673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:56.661295 18673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:56.661365 18673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:56.661410 18673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:53:56.662958 18673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:37429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:39673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:56.663228 18673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:56.663455 18673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:56.668465 18681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:56.668478 18678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:56.668465 18679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:56.668716 18673 server_base.cc:1034] running on GCE node
I20260928 22:53:56.669020 18673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:56.669215 18673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:56.670354 18673 hybrid_clock.cc:648] HybridClock initialized: now 1790636036670335 us; error 34 us; skew 500 ppm
Sep 28 22:53:56 dist-test-slave-kzfr krb5kdc[18434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:56.672777 18673 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:53:56.673079 18673 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:56.673121 18673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:56.673209 18673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:56.674331 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "505bee30a60449129aced8308cd1ec59"
format_stamp: "Formatted at 2026-09-28 22:53:56 on dist-test-slave-kzfr"
server_key: "83bf0309a4a2b645f066f65a32e03c1b"
server_key_iv: "67a154c0ac05c95d75e42e7a7949da40"
server_key_version: "encryptionkey@0"
I20260928 22:53:56.674712 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "505bee30a60449129aced8308cd1ec59"
format_stamp: "Formatted at 2026-09-28 22:53:56 on dist-test-slave-kzfr"
server_key: "83bf0309a4a2b645f066f65a32e03c1b"
server_key_iv: "67a154c0ac05c95d75e42e7a7949da40"
server_key_version: "encryptionkey@0"
I20260928 22:53:56.676069 18673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:53:56.676793 18687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:56.676982 18673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:56.677073 18673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "505bee30a60449129aced8308cd1ec59"
format_stamp: "Formatted at 2026-09-28 22:53:56 on dist-test-slave-kzfr"
server_key: "83bf0309a4a2b645f066f65a32e03c1b"
server_key_iv: "67a154c0ac05c95d75e42e7a7949da40"
server_key_version: "encryptionkey@0"
I20260928 22:53:56.677155 18673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:56.694205 18673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:56.696094 18673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:56.696262 18673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:56.696506 18673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:56.696831 18673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:56.696893 18673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:56.696939 18673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:56.696964 18673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:56.702793 18673 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:34467
I20260928 22:53:56.702831 18800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:34467 every 8 connection(s)
I20260928 22:53:56.703141 18673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:53:56.705147 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 18673
I20260928 22:53:56.705263 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:53:56.705437 14394 external_mini_cluster.cc:1428] Setting key a99529238e889c6fda4cdc7018ca1631
Sep 28 22:53:56 dist-test-slave-kzfr krb5kdc[18434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:56.713726 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.703902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:40823 (local address 127.14.14.190:39673)
0928 22:53:56.703972 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:56.703974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:56.704181 (+ 207us) server_negotiation.cc:407] Connection header received
0928 22:53:56.704727 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:56.704728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:56.704740 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:56.704792 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:56.705969 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.706520 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.708084 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.708305 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.708379 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.710608 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:56.710611 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:56.710611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:56.710619 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:56.712298 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:56.712687 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:56.712688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:56.712689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:56.712756 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:56.712998 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:56.712999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:56.712999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:56.713152 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:56.713232 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:56.713508 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 22:53:56.713624 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:53:56.713824 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.703872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:39673 (local address 127.14.14.130:40823)
0928 22:53:56.704057 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:56.704100 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:56.704612 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:56.704859 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:56.704863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:56.705153 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:56.705880 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.705887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.706613 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.706614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.707980 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.707983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.708388 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.708389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.708521 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.708949 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:56.708959 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:56.710515 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:56.712384 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:56.712390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:56.712392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:56.712611 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:56.712829 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:56.712829 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:56.712829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:56.712929 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:56.713232 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:56.713237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:56.713368 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 22:53:56.713454 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 22:53:56.713572 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260928 22:53:56.714326 18801 heartbeater.cc:347] Connected to a master server at 127.14.14.190:39673
I20260928 22:53:56.714450 18801 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:56.714599 18801 heartbeater.cc:511] Master 127.14.14.190:39673 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
I20260928 22:53:56.715003 18478 ts_manager.cc:194] Registered new tserver with Master: 505bee30a60449129aced8308cd1ec59 (127.14.14.130:34467)
I20260928 22:53:56.715377 18478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:40823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:53:56.732602 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:39673
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:37429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:56.813437 18806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:56.813663 18806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:56.813687 18806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:56.814862 18806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:56.814906 18806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:56.814961 18806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:53:56.816462 18806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:37429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:39673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:56.816699 18806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:56.816942 18806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:56.822014 18812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:56.822012 18814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:56.822014 18811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:56.822196 18806 server_base.cc:1034] running on GCE node
I20260928 22:53:56.822552 18806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:56.822808 18806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:56.823978 18806 hybrid_clock.cc:648] HybridClock initialized: now 1790636036823955 us; error 41 us; skew 500 ppm
Sep 28 22:53:56 dist-test-slave-kzfr krb5kdc[18434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:56.826480 18806 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:53:56.826874 18806 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:56.826948 18806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:56.827098 18806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:56.828174 18806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "26c9404f003c44db9a3062b79b87633d"
format_stamp: "Formatted at 2026-09-28 22:53:56 on dist-test-slave-kzfr"
server_key: "828cd438fe2b840148d2ad73a14b04e8"
server_key_iv: "052171d7c209de4e71cf10a9c00e3aac"
server_key_version: "encryptionkey@0"
I20260928 22:53:56.828544 18806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "26c9404f003c44db9a3062b79b87633d"
format_stamp: "Formatted at 2026-09-28 22:53:56 on dist-test-slave-kzfr"
server_key: "828cd438fe2b840148d2ad73a14b04e8"
server_key_iv: "052171d7c209de4e71cf10a9c00e3aac"
server_key_version: "encryptionkey@0"
I20260928 22:53:56.829910 18806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:53:56.830662 18820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:56.830886 18806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:53:56.830994 18806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "26c9404f003c44db9a3062b79b87633d"
format_stamp: "Formatted at 2026-09-28 22:53:56 on dist-test-slave-kzfr"
server_key: "828cd438fe2b840148d2ad73a14b04e8"
server_key_iv: "052171d7c209de4e71cf10a9c00e3aac"
server_key_version: "encryptionkey@0"
I20260928 22:53:56.831070 18806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:56.869375 18806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:56.871294 18806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:56.871464 18806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:56.871708 18806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:56.872021 18806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:56.872080 18806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:56.872126 18806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:56.872159 18806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:56.878311 18806 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:33359
I20260928 22:53:56.878346 18933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:33359 every 8 connection(s)
I20260928 22:53:56.878613 18806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:53:56 dist-test-slave-kzfr krb5kdc[18434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:53:56.887468 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.879295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:40469 (local address 127.14.14.190:39673)
0928 22:53:56.879361 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:56.879362 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:56.879566 (+ 204us) server_negotiation.cc:407] Connection header received
0928 22:53:56.880251 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:56.880251 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:56.880262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:56.880301 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:56.881320 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.881875 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.882916 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.883097 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.883150 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.885149 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:56.885149 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:56.885150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:56.885155 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:56.886381 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:56.886725 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:56.886725 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:56.886726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:56.886775 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:56.886986 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:56.886986 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:56.886986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:56.887083 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:56.887133 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:56.887349 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 22:53:56.887406 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:53:56.887633 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 18806
I20260928 22:53:56.887703 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.879281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:39673 (local address 127.14.14.131:40469)
0928 22:53:56.879454 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:56.879488 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:56.880122 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:56.880397 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:56.880401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:56.880704 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:56.881244 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.881250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.881949 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.881950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.882841 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.882850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.883169 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.883169 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.883265 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.883766 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:56.883774 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:56.885076 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:56.886451 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:56.886456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:56.886458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:56.886661 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:56.886830 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:56.886831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:56.886831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:56.886933 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:56.887135 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:56.887139 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:56.887245 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 22:53:56.887324 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 22:53:56.887434 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260928 22:53:56.887739 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:53:56.887939 14394 external_mini_cluster.cc:1428] Setting key a8a6fe12d401ae2b62f887598b612ec2
I20260928 22:53:56.888180 18934 heartbeater.cc:347] Connected to a master server at 127.14.14.190:39673
I20260928 22:53:56.888283 18934 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:56.888442 18934 heartbeater.cc:511] Master 127.14.14.190:39673 requested a full tablet report, sending...
I20260928 22:53:56.888861 18478 ts_manager.cc:194] Registered new tserver with Master: 26c9404f003c44db9a3062b79b87633d (127.14.14.131:33359)
I20260928 22:53:56.889209 18478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:40469
I20260928 22:53:56.899794 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:53:56.906702 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.901140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48838 (local address 127.14.14.190:39673)
0928 22:53:56.901182 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:56.901186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:56.901242 (+ 56us) server_negotiation.cc:407] Connection header received
0928 22:53:56.901294 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:56.901295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:56.901303 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:56.901355 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:56.901952 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.902425 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.903109 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.903276 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.903323 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.904403 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:56.904404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:56.904404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:56.904410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:56.905817 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:56.906115 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:56.906115 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:56.906116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:56.906165 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:56.906309 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:56.906309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:56.906309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:56.906434 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:56.906504 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:56.906583 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 22:53:56.906640 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:53:56.908412 18478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:53:56.910002 18478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:53:56.916584 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.912249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:33359 (local address 127.0.0.1:47334)
0928 22:53:56.912560 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:56.912593 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:56.912670 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:56.913073 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:56.913074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:56.913077 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:56.913130 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.913132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.914408 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.914409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.915569 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.915571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.916280 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.916281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.916361 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.916377 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 22:53:56.916450 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:53:56.916509 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":74,"threads_started":1}
I20260928 22:53:56.916596 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.912359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47334 (local address 127.14.14.131:33359)
0928 22:53:56.912614 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:56.912617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:56.912841 (+ 224us) server_negotiation.cc:407] Connection header received
0928 22:53:56.912897 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:56.912899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:56.912916 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:56.912985 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:56.913215 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.914319 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.915649 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.916191 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.916281 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.916352 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:56.916468 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 22:53:56.916510 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":107,"threads_started":1}
I20260928 22:53:56.917773 18868 tablet_service.cc:1467] Processing CreateTablet for tablet 740e34229e2849d596492c8e8e647266 (DEFAULT_TABLE table=test-table [id=eb21125669bb45d99a2c1c67260364e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:56.918008 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.912242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:34467 (local address 127.0.0.1:48570)
0928 22:53:56.912542 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:56.912585 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:56.912683 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:56.912952 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:56.912956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:56.912958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:56.913028 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.913030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.915654 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.915655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.916849 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.916861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.917746 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.917747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.917826 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.917830 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:56.917893 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 22:53:56.917934 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260928 22:53:56.918025 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.912345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.14.14.130:34467)
0928 22:53:56.912535 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:56.912540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:56.912652 (+ 112us) server_negotiation.cc:407] Connection header received
0928 22:53:56.912814 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:56.912818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:56.912835 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:56.912945 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:56.913149 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.914309 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.916938 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.917665 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.917735 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.917786 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:56.917910 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 22:53:56.917947 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260928 22:53:56.918180 18868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740e34229e2849d596492c8e8e647266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:56.918695 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.912498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:41789 (local address 127.0.0.1:58782)
0928 22:53:56.912697 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:56.912706 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:56.912746 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:56.914642 (+ 1896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:56.914642 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:56.914644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:56.914692 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.914693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.915734 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.915734 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.916850 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.916852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.918457 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.918458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.918528 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.918532 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:56.918588 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:53:56.918627 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":23,"thread_start_us":66,"threads_started":1}
I20260928 22:53:56.918709 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.912702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.14.14.129:41789)
0928 22:53:56.914500 (+ 1798us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:56.914504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:56.914511 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:53:56.914541 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:56.914542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:56.914554 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:56.914614 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:56.914770 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.915635 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.917640 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.918358 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.918453 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.918504 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:56.918600 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 22:53:56.918636 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1755,"thread_start_us":76,"threads_started":1}
I20260928 22:53:56.918664 18735 tablet_service.cc:1467] Processing CreateTablet for tablet 740e34229e2849d596492c8e8e647266 (DEFAULT_TABLE table=test-table [id=eb21125669bb45d99a2c1c67260364e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:56.918974 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740e34229e2849d596492c8e8e647266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:56.919258 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 740e34229e2849d596492c8e8e647266 (DEFAULT_TABLE table=test-table [id=eb21125669bb45d99a2c1c67260364e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:56.919484 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740e34229e2849d596492c8e8e647266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:56.921634 18951 tablet_bootstrap.cc:492] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d: Bootstrap starting.
I20260928 22:53:56.922305 18953 tablet_bootstrap.cc:492] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59: Bootstrap starting.
I20260928 22:53:56.922384 18951 tablet_bootstrap.cc:654] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:56.923096 18953 tablet_bootstrap.cc:654] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:56.923087 18951 log.cc:824] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:56.923467 18952 tablet_bootstrap.cc:492] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b: Bootstrap starting.
I20260928 22:53:56.923694 18953 log.cc:824] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:56.923897 18951 tablet_bootstrap.cc:492] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d: No bootstrap required, opened a new log
I20260928 22:53:56.924014 18951 ts_tablet_manager.cc:1397] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:53:56.924129 18952 tablet_bootstrap.cc:654] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:56.924563 18953 tablet_bootstrap.cc:492] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59: No bootstrap required, opened a new log
I20260928 22:53:56.924646 18953 ts_tablet_manager.cc:1397] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:53:56.924721 18952 log.cc:824] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:56.925513 18952 tablet_bootstrap.cc:492] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b: No bootstrap required, opened a new log
I20260928 22:53:56.925606 18952 ts_tablet_manager.cc:1397] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:53:56.925935 18951 raft_consensus.cc:348] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505bee30a60449129aced8308cd1ec59" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34467 } } peers { permanent_uuid: "26c9404f003c44db9a3062b79b87633d" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33359 } } peers { permanent_uuid: "b6b4a3b3f4c24d17a0c6e0658448552b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41789 } }
I20260928 22:53:56.926131 18951 raft_consensus.cc:374] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:56.926198 18951 raft_consensus.cc:729] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c9404f003c44db9a3062b79b87633d, State: Initialized, Role: FOLLOWER
I20260928 22:53:56.926331 18951 consensus_queue.cc:260] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505bee30a60449129aced8308cd1ec59" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34467 } } peers { permanent_uuid: "26c9404f003c44db9a3062b79b87633d" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33359 } } peers { permanent_uuid: "b6b4a3b3f4c24d17a0c6e0658448552b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41789 } }
I20260928 22:53:56.926626 18951 ts_tablet_manager.cc:1428] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 22:53:56.926759 18934 heartbeater.cc:503] Master 127.14.14.190:39673 was elected leader, sending a full tablet report...
I20260928 22:53:56.926671 18953 raft_consensus.cc:348] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505bee30a60449129aced8308cd1ec59" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34467 } } peers { permanent_uuid: "26c9404f003c44db9a3062b79b87633d" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33359 } } peers { permanent_uuid: "b6b4a3b3f4c24d17a0c6e0658448552b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41789 } }
I20260928 22:53:56.926838 18953 raft_consensus.cc:374] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:56.926930 18953 raft_consensus.cc:729] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 505bee30a60449129aced8308cd1ec59, State: Initialized, Role: FOLLOWER
I20260928 22:53:56.927075 18953 consensus_queue.cc:260] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505bee30a60449129aced8308cd1ec59" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34467 } } peers { permanent_uuid: "26c9404f003c44db9a3062b79b87633d" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33359 } } peers { permanent_uuid: "b6b4a3b3f4c24d17a0c6e0658448552b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41789 } }
I20260928 22:53:56.927366 18953 ts_tablet_manager.cc:1428] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:53:56.927393 18801 heartbeater.cc:503] Master 127.14.14.190:39673 was elected leader, sending a full tablet report...
I20260928 22:53:56.927670 18952 raft_consensus.cc:348] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505bee30a60449129aced8308cd1ec59" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34467 } } peers { permanent_uuid: "26c9404f003c44db9a3062b79b87633d" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33359 } } peers { permanent_uuid: "b6b4a3b3f4c24d17a0c6e0658448552b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41789 } }
I20260928 22:53:56.927865 18952 raft_consensus.cc:374] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:56.927922 18952 raft_consensus.cc:729] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b4a3b3f4c24d17a0c6e0658448552b, State: Initialized, Role: FOLLOWER
I20260928 22:53:56.928035 18952 consensus_queue.cc:260] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505bee30a60449129aced8308cd1ec59" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34467 } } peers { permanent_uuid: "26c9404f003c44db9a3062b79b87633d" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33359 } } peers { permanent_uuid: "b6b4a3b3f4c24d17a0c6e0658448552b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41789 } }
I20260928 22:53:56.928296 18952 ts_tablet_manager.cc:1428] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:53:56.928296 18668 heartbeater.cc:503] Master 127.14.14.190:39673 was elected leader, sending a full tablet report...
I20260928 22:53:56.932349 18959 raft_consensus.cc:482] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:53:56.932421 18959 raft_consensus.cc:504] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505bee30a60449129aced8308cd1ec59" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34467 } } peers { permanent_uuid: "26c9404f003c44db9a3062b79b87633d" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33359 } } peers { permanent_uuid: "b6b4a3b3f4c24d17a0c6e0658448552b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41789 } }
I20260928 22:53:56.932641 18959 leader_election.cc:288] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 505bee30a60449129aced8308cd1ec59 (127.14.14.130:34467), 26c9404f003c44db9a3062b79b87633d (127.14.14.131:33359)
I20260928 22:53:56.935967 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.933070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:49619 (local address 127.14.14.131:33359)
0928 22:53:56.933128 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:56.933130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:56.933138 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:53:56.933160 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:56.933160 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:56.933181 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:56.933233 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:56.933438 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.934281 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.935200 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.935622 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.935716 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.935724 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:56.935872 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 22:53:56.935906 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:53:56.935984 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.932806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:33359 (local address 127.14.14.129:49619)
0928 22:53:56.932952 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:56.932960 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:56.932999 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:56.933249 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:56.933250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:56.933251 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:56.933307 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.933308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.934367 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.934368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.935097 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.935099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.935717 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.935718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.935787 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.935791 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:56.935867 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 22:53:56.935905 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":21,"thread_start_us":72,"threads_started":1}
I20260928 22:53:56.936067 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.932870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:48773 (local address 127.14.14.130:34467)
0928 22:53:56.932922 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:56.932924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:56.932937 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:53:56.933015 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:56.933016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:56.933024 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:56.933097 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:56.933366 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.934165 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.935281 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.935683 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.935770 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.935778 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:56.935956 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 22:53:56.935991 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:53:56.936239 18888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740e34229e2849d596492c8e8e647266" candidate_uuid: "b6b4a3b3f4c24d17a0c6e0658448552b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c9404f003c44db9a3062b79b87633d" is_pre_election: true
I20260928 22:53:56.936334 18888 raft_consensus.cc:2454] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6b4a3b3f4c24d17a0c6e0658448552b in term 0.
I20260928 22:53:56.936426 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.932807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:34467 (local address 127.14.14.129:48773)
0928 22:53:56.932896 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:56.932908 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:56.932950 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:56.933173 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:56.933174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:56.933176 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:56.933271 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.933273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.934248 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.934249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.935190 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:56.935193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.935768 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.935769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:56.935868 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.935871 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:53:56.935948 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 22:53:56.935986 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":1}
I20260928 22:53:56.936609 18558 leader_election.cc:302] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c9404f003c44db9a3062b79b87633d, b6b4a3b3f4c24d17a0c6e0658448552b; no voters:
I20260928 22:53:56.936720 18959 raft_consensus.cc:2781] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:53:56.936724 18755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740e34229e2849d596492c8e8e647266" candidate_uuid: "b6b4a3b3f4c24d17a0c6e0658448552b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "505bee30a60449129aced8308cd1ec59" is_pre_election: true
I20260928 22:53:56.936785 18959 raft_consensus.cc:482] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:53:56.936825 18959 raft_consensus.cc:3037] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:56.936858 18755 raft_consensus.cc:2454] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6b4a3b3f4c24d17a0c6e0658448552b in term 0.
I20260928 22:53:56.937496 18959 raft_consensus.cc:504] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505bee30a60449129aced8308cd1ec59" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34467 } } peers { permanent_uuid: "26c9404f003c44db9a3062b79b87633d" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33359 } } peers { permanent_uuid: "b6b4a3b3f4c24d17a0c6e0658448552b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41789 } }
I20260928 22:53:56.937667 18959 leader_election.cc:288] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [CANDIDATE]: Term 1 election: Requested vote from peers 505bee30a60449129aced8308cd1ec59 (127.14.14.130:34467), 26c9404f003c44db9a3062b79b87633d (127.14.14.131:33359)
I20260928 22:53:56.937783 18755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740e34229e2849d596492c8e8e647266" candidate_uuid: "b6b4a3b3f4c24d17a0c6e0658448552b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "505bee30a60449129aced8308cd1ec59"
I20260928 22:53:56.937806 18888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740e34229e2849d596492c8e8e647266" candidate_uuid: "b6b4a3b3f4c24d17a0c6e0658448552b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c9404f003c44db9a3062b79b87633d"
I20260928 22:53:56.937855 18755 raft_consensus.cc:3037] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:56.937870 18888 raft_consensus.cc:3037] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:56.938674 18888 raft_consensus.cc:2454] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6b4a3b3f4c24d17a0c6e0658448552b in term 1.
I20260928 22:53:56.938674 18755 raft_consensus.cc:2454] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6b4a3b3f4c24d17a0c6e0658448552b in term 1.
I20260928 22:53:56.938830 18558 leader_election.cc:302] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c9404f003c44db9a3062b79b87633d, b6b4a3b3f4c24d17a0c6e0658448552b; no voters:
I20260928 22:53:56.938957 18959 raft_consensus.cc:2781] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:56.939078 18959 raft_consensus.cc:686] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [term 1 LEADER]: Becoming Leader. State: Replica: b6b4a3b3f4c24d17a0c6e0658448552b, State: Running, Role: LEADER
I20260928 22:53:56.939174 18959 consensus_queue.cc:237] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505bee30a60449129aced8308cd1ec59" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34467 } } peers { permanent_uuid: "26c9404f003c44db9a3062b79b87633d" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33359 } } peers { permanent_uuid: "b6b4a3b3f4c24d17a0c6e0658448552b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41789 } }
I20260928 22:53:56.939885 18477 catalog_manager.cc:5495] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b reported cstate change: term changed from 0 to 1, leader changed from <none> to b6b4a3b3f4c24d17a0c6e0658448552b (127.14.14.129). New cstate: current_term: 1 leader_uuid: "b6b4a3b3f4c24d17a0c6e0658448552b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505bee30a60449129aced8308cd1ec59" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26c9404f003c44db9a3062b79b87633d" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b4a3b3f4c24d17a0c6e0658448552b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41789 } health_report { overall_health: HEALTHY } } }
I20260928 22:53:56.945349 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.943217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.14.14.129:41789)
0928 22:53:56.943252 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:56.943253 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:56.943261 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:53:56.943329 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:56.943329 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:56.943343 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:56.943404 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:56.943523 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.943948 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.944750 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:56.944929 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:56.944980 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:56.945104 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:56.945189 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:56.945246 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:56.945277 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 22:53:56.945291 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 22:53:56.947273 18755 raft_consensus.cc:1261] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59 [term 1 FOLLOWER]: Refusing update from remote peer b6b4a3b3f4c24d17a0c6e0658448552b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:56.947285 18888 raft_consensus.cc:1261] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d [term 1 FOLLOWER]: Refusing update from remote peer b6b4a3b3f4c24d17a0c6e0658448552b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:56.947609 18959 consensus_queue.cc:1035] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c9404f003c44db9a3062b79b87633d" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:56.947808 18961 consensus_queue.cc:1035] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b [LEADER]: Connected to new peer: Peer: permanent_uuid: "505bee30a60449129aced8308cd1ec59" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:56.949640 18964 mvcc.cc:205] Tried to move back new op lower bound from 7334445207332564992 to 7334445207303434240. Current Snapshot: MvccSnapshot[applied={T|T < 7334445207332564992}]
I20260928 22:53:56.950568 18965 mvcc.cc:205] Tried to move back new op lower bound from 7334445207332564992 to 7334445207303434240. Current Snapshot: MvccSnapshot[applied={T|T < 7334445207332564992 or (T in {7334445207332564992})}]
I20260928 22:53:56.951988 18478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:56.952098 18478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:56.952852 18478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8471e661291436f9e726369246fa280: Sending DeleteTablet for 3 replicas of tablet 740e34229e2849d596492c8e8e647266
I20260928 22:53:56.953191 18868 tablet_service.cc:1514] Processing DeleteTablet for tablet 740e34229e2849d596492c8e8e647266 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:56 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:47334
I20260928 22:53:56.953194 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 740e34229e2849d596492c8e8e647266 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:56 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:58782
I20260928 22:53:56.953455 18973 tablet_replica.cc:330] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d: stopping tablet replica
I20260928 22:53:56.953444 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 740e34229e2849d596492c8e8e647266 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:56 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:48570
I20260928 22:53:56.953660 18975 tablet_replica.cc:330] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59: stopping tablet replica
I20260928 22:53:56.953740 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 18540
I20260928 22:53:56.953791 18974 tablet_replica.cc:330] T 740e34229e2849d596492c8e8e647266 P b6b4a3b3f4c24d17a0c6e0658448552b: stopping tablet replica
I20260928 22:53:56.954579 18975 raft_consensus.cc:2229] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:56.954685 18975 raft_consensus.cc:2258] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:56.955099 18975 ts_tablet_manager.cc:1905] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:56.956570 18975 ts_tablet_manager.cc:1918] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:53:56.956629 18975 log.cc:1196] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/740e34229e2849d596492c8e8e647266
I20260928 22:53:56.956914 18975 ts_tablet_manager.cc:1939] T 740e34229e2849d596492c8e8e647266 P 505bee30a60449129aced8308cd1ec59: Deleting consensus metadata
I20260928 22:53:56.957274 18462 catalog_manager.cc:4841] TS 505bee30a60449129aced8308cd1ec59 (127.14.14.130:34467): tablet 740e34229e2849d596492c8e8e647266 (table test-table [id=eb21125669bb45d99a2c1c67260364e2]) successfully deleted
W20260928 22:53:56.959668 18462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:41789 (error 108)
I20260928 22:53:56.959689 18973 raft_consensus.cc:2229] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:56.959813 18973 raft_consensus.cc:2258] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:56.959870 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 18673
I20260928 22:53:56.961061 18973 ts_tablet_manager.cc:1905] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:56.961620 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:56.961460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:41789 (local address 127.0.0.1:58798)
0928 22:53:56.961508 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:56.961547 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:41789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 22:53:56.961760 18462 catalog_manager.cc:4584] TS b6b4a3b3f4c24d17a0c6e0658448552b (127.14.14.129:41789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 740e34229e2849d596492c8e8e647266: Network error: Client connection negotiation failed: client connection to 127.14.14.129:41789: connect: Connection refused (error 111)
I20260928 22:53:56.963079 18973 ts_tablet_manager.cc:1918] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:53:56.963164 18973 log.cc:1196] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/740e34229e2849d596492c8e8e647266
I20260928 22:53:56.963428 18973 ts_tablet_manager.cc:1939] T 740e34229e2849d596492c8e8e647266 P 26c9404f003c44db9a3062b79b87633d: Deleting consensus metadata
I20260928 22:53:56.963848 18465 catalog_manager.cc:4841] TS 26c9404f003c44db9a3062b79b87633d (127.14.14.131:33359): tablet 740e34229e2849d596492c8e8e647266 (table test-table [id=eb21125669bb45d99a2c1c67260364e2]) successfully deleted
I20260928 22:53:56.965267 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 18806
I20260928 22:53:56.969542 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 18447
2026-09-28T22:53:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1889 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T22:53:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:56Z Disabled control of system clock
I20260928 22:53:56.996590 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:45925
--webserver_interface=127.14.14.190
--webserver_port=0
--builtin_ntp_servers=127.14.14.148:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:45925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 22:53:57.077139 18980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:57.077355 18980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:57.077378 18980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:57.078455 18980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:57.078491 18980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:57.078504 18980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:57.078517 18980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:57.078529 18980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:57.079824 18980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:45925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:45925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:57.080010 18980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:57.080312 18980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:57.085311 18985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:57.085305 18988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:57.085320 18986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:57.085563 18980 server_base.cc:1034] running on GCE node
I20260928 22:53:57.085836 18980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:57.086158 18980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:57.087332 18980 hybrid_clock.cc:648] HybridClock initialized: now 1790636037087314 us; error 51 us; skew 500 ppm
I20260928 22:53:57.088737 18980 webserver.cc:460] Webserver started at http://127.14.14.190:34849/ using document root <none> and password file <none>
I20260928 22:53:57.088979 18980 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:57.089051 18980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:57.089211 18980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:57.090363 18980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "fe1f6d6eece14cddb72d1da5f2ce0d98"
format_stamp: "Formatted at 2026-09-28 22:53:57 on dist-test-slave-kzfr"
server_key: "4a2ccc635f4c2897486cd971c8f9269e"
server_key_iv: "055586953826be1c8078efda277926bd"
server_key_version: "encryptionkey@0"
I20260928 22:53:57.090726 18980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "fe1f6d6eece14cddb72d1da5f2ce0d98"
format_stamp: "Formatted at 2026-09-28 22:53:57 on dist-test-slave-kzfr"
server_key: "4a2ccc635f4c2897486cd971c8f9269e"
server_key_iv: "055586953826be1c8078efda277926bd"
server_key_version: "encryptionkey@0"
I20260928 22:53:57.091986 18980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:53:57.092665 18994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:57.092841 18980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:57.092942 18980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "fe1f6d6eece14cddb72d1da5f2ce0d98"
format_stamp: "Formatted at 2026-09-28 22:53:57 on dist-test-slave-kzfr"
server_key: "4a2ccc635f4c2897486cd971c8f9269e"
server_key_iv: "055586953826be1c8078efda277926bd"
server_key_version: "encryptionkey@0"
I20260928 22:53:57.093031 18980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:57.096307 18980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:57.096565 18980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:57.096704 18980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:57.100538 18980 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:45925
I20260928 22:53:57.100586 19046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:45925 every 8 connection(s)
I20260928 22:53:57.100944 18980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:57.101670 19047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:57.104314 19047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98: Bootstrap starting.
I20260928 22:53:57.104899 19047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:57.105192 19047 log.cc:824] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:57.105897 19047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98: No bootstrap required, opened a new log
I20260928 22:53:57.107424 19047 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f6d6eece14cddb72d1da5f2ce0d98" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 45925 } }
I20260928 22:53:57.107554 19047 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:57.107576 19047 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1f6d6eece14cddb72d1da5f2ce0d98, State: Initialized, Role: FOLLOWER
I20260928 22:53:57.107748 19047 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f6d6eece14cddb72d1da5f2ce0d98" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 45925 } }
I20260928 22:53:57.107807 19047 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:57.107828 19047 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:57.107898 19047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:57.108421 19047 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f6d6eece14cddb72d1da5f2ce0d98" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 45925 } }
I20260928 22:53:57.108534 19047 leader_election.cc:302] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe1f6d6eece14cddb72d1da5f2ce0d98; no voters:
I20260928 22:53:57.108711 19047 leader_election.cc:288] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:57.108744 19050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:57.108877 19050 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [term 1 LEADER]: Becoming Leader. State: Replica: fe1f6d6eece14cddb72d1da5f2ce0d98, State: Running, Role: LEADER
I20260928 22:53:57.108981 19047 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:57.109018 19050 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f6d6eece14cddb72d1da5f2ce0d98" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 45925 } }
I20260928 22:53:57.109552 19051 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe1f6d6eece14cddb72d1da5f2ce0d98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f6d6eece14cddb72d1da5f2ce0d98" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 45925 } } }
I20260928 22:53:57.109668 19051 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:57.109819 19052 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe1f6d6eece14cddb72d1da5f2ce0d98. Latest consensus state: current_term: 1 leader_uuid: "fe1f6d6eece14cddb72d1da5f2ce0d98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f6d6eece14cddb72d1da5f2ce0d98" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 45925 } } }
I20260928 22:53:57.109900 19052 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:57.109926 19059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:57.110481 19059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:57.110759 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 18980
I20260928 22:53:57.110863 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:53:57.111042 14394 external_mini_cluster.cc:1428] Setting key 6006e649756602bd6246f35be2d30cb4
I20260928 22:53:57.112237 19059 catalog_manager.cc:1347] Generated new cluster ID: b842d6f196cd4150959c6943c4974480
I20260928 22:53:57.112330 19059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:57.113528 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.111570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43154 (local address 127.14.14.190:45925)
0928 22:53:57.111945 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:57.111953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:57.111977 (+ 24us) server_negotiation.cc:407] Connection header received
0928 22:53:57.112067 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:57.112072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:57.112128 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:57.112310 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:57.112998 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:57.113001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:57.113001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:57.113005 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:57.113046 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:57.113059 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:57.113121 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:57.113227 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 22:53:57.113364 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":67,"threads_started":1}
I20260928 22:53:57.124548 19059 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:57.124744 19059 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:57.136339 19059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98: Generated new TSK 0
I20260928 22:53:57.136512 19059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:57.138198 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:45925
--builtin_ntp_servers=127.14.14.148:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 22:53:57.218657 19071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:57.218921 19071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:57.218961 19071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:57.220139 19071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:57.220204 19071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:57.220249 19071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:53:57.221669 19071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:45925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:57.221931 19071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:57.222231 19071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:57.227486 19077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:57.227511 19071 server_base.cc:1034] running on GCE node
W20260928 22:53:57.227626 19079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:57.227490 19076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:57.227943 19071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:57.228207 19071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:57.229405 19071 hybrid_clock.cc:648] HybridClock initialized: now 1790636037229380 us; error 39 us; skew 500 ppm
I20260928 22:53:57.230794 19071 webserver.cc:460] Webserver started at http://127.14.14.129:38631/ using document root <none> and password file <none>
I20260928 22:53:57.231041 19071 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:57.231086 19071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:57.231231 19071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:57.232290 19071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "5a4528c147d94db98946565c9abb93c5"
format_stamp: "Formatted at 2026-09-28 22:53:57 on dist-test-slave-kzfr"
server_key: "ed18cd1fb93c08c65d7e9cdba085718f"
server_key_iv: "52bcc76fc271d0d3f3e100416dbd9fc2"
server_key_version: "encryptionkey@0"
I20260928 22:53:57.232657 19071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "5a4528c147d94db98946565c9abb93c5"
format_stamp: "Formatted at 2026-09-28 22:53:57 on dist-test-slave-kzfr"
server_key: "ed18cd1fb93c08c65d7e9cdba085718f"
server_key_iv: "52bcc76fc271d0d3f3e100416dbd9fc2"
server_key_version: "encryptionkey@0"
I20260928 22:53:57.234035 19071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:53:57.234809 19085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:57.234964 19071 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:53:57.235047 19071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "5a4528c147d94db98946565c9abb93c5"
format_stamp: "Formatted at 2026-09-28 22:53:57 on dist-test-slave-kzfr"
server_key: "ed18cd1fb93c08c65d7e9cdba085718f"
server_key_iv: "52bcc76fc271d0d3f3e100416dbd9fc2"
server_key_version: "encryptionkey@0"
I20260928 22:53:57.235114 19071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:57.238452 19071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:57.238731 19071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:57.238865 19071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:57.239075 19071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:57.239370 19071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:57.239401 19071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:57.239459 19071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:57.239485 19071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:57.245203 19071 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:36947
I20260928 22:53:57.245252 19198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:36947 every 8 connection(s)
I20260928 22:53:57.245548 19071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:53:57.248232 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.246284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:44023 (local address 127.14.14.190:45925)
0928 22:53:57.246346 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:57.246353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:57.246560 (+ 207us) server_negotiation.cc:407] Connection header received
0928 22:53:57.246977 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:57.246978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:57.246983 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:57.247039 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:57.248050 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:57.248051 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:57.248051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:57.248055 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:57.248069 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:57.248073 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:57.248118 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:57.248169 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 22:53:57.248182 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:53:57.248438 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.246271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:45925 (local address 127.14.14.129:44023)
0928 22:53:57.246437 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:57.246476 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:57.246901 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:57.247086 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:57.247089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:57.247096 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:57.247919 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:53:57.247920 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:57.247945 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:53:57.247970 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:57.248116 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:57.248118 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:57.248120 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:53:57.248169 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 22:53:57.248213 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260928 22:53:57.249009 19199 heartbeater.cc:347] Connected to a master server at 127.14.14.190:45925
I20260928 22:53:57.249104 19199 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:57.249248 19199 heartbeater.cc:511] Master 127.14.14.190:45925 requested a full tablet report, sending...
I20260928 22:53:57.249760 19011 ts_manager.cc:194] Registered new tserver with Master: 5a4528c147d94db98946565c9abb93c5 (127.14.14.129:36947)
I20260928 22:53:57.253696 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 19071
I20260928 22:53:57.253829 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:53:57.254047 14394 external_mini_cluster.cc:1428] Setting key c732e735931622ec7754b6f18aaf5ba5
I20260928 22:53:57.255007 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:45925
--builtin_ntp_servers=127.14.14.148:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 22:53:57.343057 19202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:57.343277 19202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:57.343302 19202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:57.344563 19202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:57.344606 19202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:57.344671 19202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:53:57.346215 19202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:45925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:57.346472 19202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:57.346778 19202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:57.351953 19207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:57.352008 19208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:57.351987 19202 server_base.cc:1034] running on GCE node
W20260928 22:53:57.351963 19210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:57.352298 19202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:57.352522 19202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:57.353628 19202 hybrid_clock.cc:648] HybridClock initialized: now 1790636037353613 us; error 28 us; skew 500 ppm
I20260928 22:53:57.355157 19202 webserver.cc:460] Webserver started at http://127.14.14.130:38089/ using document root <none> and password file <none>
I20260928 22:53:57.355405 19202 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:57.355477 19202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:57.355611 19202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:57.356644 19202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "6e2d583d7a9b4a1191d21904d0969ea8"
format_stamp: "Formatted at 2026-09-28 22:53:57 on dist-test-slave-kzfr"
server_key: "fbfa383ee5e991f54f9bb6903f733160"
server_key_iv: "19d0234b3a3fe4390630e41ed8c8487b"
server_key_version: "encryptionkey@0"
I20260928 22:53:57.357010 19202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "6e2d583d7a9b4a1191d21904d0969ea8"
format_stamp: "Formatted at 2026-09-28 22:53:57 on dist-test-slave-kzfr"
server_key: "fbfa383ee5e991f54f9bb6903f733160"
server_key_iv: "19d0234b3a3fe4390630e41ed8c8487b"
server_key_version: "encryptionkey@0"
I20260928 22:53:57.358358 19202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:57.359026 19216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:57.359196 19202 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:57.359294 19202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "6e2d583d7a9b4a1191d21904d0969ea8"
format_stamp: "Formatted at 2026-09-28 22:53:57 on dist-test-slave-kzfr"
server_key: "fbfa383ee5e991f54f9bb6903f733160"
server_key_iv: "19d0234b3a3fe4390630e41ed8c8487b"
server_key_version: "encryptionkey@0"
I20260928 22:53:57.359369 19202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:57.362679 19202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:57.362923 19202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:57.363055 19202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:57.363246 19202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:57.363533 19202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:57.363583 19202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:57.363626 19202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:57.363654 19202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:57.369063 19202 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:41843
I20260928 22:53:57.369149 19329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:41843 every 8 connection(s)
I20260928 22:53:57.369444 19202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:53:57.372171 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.370165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:33981 (local address 127.14.14.190:45925)
0928 22:53:57.370234 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:57.370237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:57.370468 (+ 231us) server_negotiation.cc:407] Connection header received
0928 22:53:57.370882 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:57.370883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:57.370889 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:57.370941 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:57.371996 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:57.371997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:57.371997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:57.372001 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:57.372016 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:57.372020 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:57.372066 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:57.372110 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 22:53:57.372121 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:53:57.372351 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.370156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:45925 (local address 127.14.14.130:33981)
0928 22:53:57.370340 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:57.370376 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:57.370803 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:57.370989 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:57.370992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:57.371000 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:57.371852 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:53:57.371853 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:57.371892 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:53:57.371918 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:57.372063 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:57.372065 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:57.372067 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:53:57.372116 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 22:53:57.372159 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260928 22:53:57.372803 19330 heartbeater.cc:347] Connected to a master server at 127.14.14.190:45925
I20260928 22:53:57.372902 19330 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:57.373040 19330 heartbeater.cc:511] Master 127.14.14.190:45925 requested a full tablet report, sending...
I20260928 22:53:57.373380 19011 ts_manager.cc:194] Registered new tserver with Master: 6e2d583d7a9b4a1191d21904d0969ea8 (127.14.14.130:41843)
I20260928 22:53:57.379673 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 19202
I20260928 22:53:57.379762 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:53:57.379948 14394 external_mini_cluster.cc:1428] Setting key d1d01214cfc3bbdf65b19cba15591b4a
I20260928 22:53:57.380885 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:45925
--builtin_ntp_servers=127.14.14.148:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 22:53:57.461843 19333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:57.462055 19333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:57.462078 19333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:57.463376 19333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:57.463419 19333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:57.463446 19333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:53:57.464777 19333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:45925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:57.465041 19333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:57.465332 19333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:57.470532 19339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:57.470553 19338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:57.470556 19341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:57.470593 19333 server_base.cc:1034] running on GCE node
I20260928 22:53:57.470999 19333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:57.471231 19333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:57.472384 19333 hybrid_clock.cc:648] HybridClock initialized: now 1790636037472372 us; error 46 us; skew 500 ppm
I20260928 22:53:57.473650 19333 webserver.cc:460] Webserver started at http://127.14.14.131:40721/ using document root <none> and password file <none>
I20260928 22:53:57.473898 19333 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:57.473946 19333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:57.474080 19333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:57.475193 19333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "33cd5195f64941e6986c6afd3120c8db"
format_stamp: "Formatted at 2026-09-28 22:53:57 on dist-test-slave-kzfr"
server_key: "2f8f7c0bb5e92018ae48227ea0348ff3"
server_key_iv: "2f9508aa111a43e74d39d8d55dbc79c1"
server_key_version: "encryptionkey@0"
I20260928 22:53:57.475559 19333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "33cd5195f64941e6986c6afd3120c8db"
format_stamp: "Formatted at 2026-09-28 22:53:57 on dist-test-slave-kzfr"
server_key: "2f8f7c0bb5e92018ae48227ea0348ff3"
server_key_iv: "2f9508aa111a43e74d39d8d55dbc79c1"
server_key_version: "encryptionkey@0"
I20260928 22:53:57.476919 19333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:57.477759 19347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:57.477977 19333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:57.478065 19333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "33cd5195f64941e6986c6afd3120c8db"
format_stamp: "Formatted at 2026-09-28 22:53:57 on dist-test-slave-kzfr"
server_key: "2f8f7c0bb5e92018ae48227ea0348ff3"
server_key_iv: "2f9508aa111a43e74d39d8d55dbc79c1"
server_key_version: "encryptionkey@0"
I20260928 22:53:57.478123 19333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:57.481418 19333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:57.481710 19333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:57.481874 19333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:57.482072 19333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:57.482332 19333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:57.482364 19333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:57.482385 19333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:57.482398 19333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:57.488070 19333 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:43349
I20260928 22:53:57.488152 19460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:43349 every 8 connection(s)
I20260928 22:53:57.488514 19333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:53:57.491710 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.489377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:43797 (local address 127.14.14.190:45925)
0928 22:53:57.489419 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:57.489421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:57.489655 (+ 234us) server_negotiation.cc:407] Connection header received
0928 22:53:57.490203 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:57.490205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:57.490211 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:57.490270 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:57.491419 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:57.491420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:57.491421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:57.491426 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:57.491443 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:57.491448 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:57.491528 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:57.491621 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 22:53:57.491637 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:53:57.491916 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.489339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:45925 (local address 127.14.14.131:43797)
0928 22:53:57.489516 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:57.489549 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:57.490089 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:57.490348 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:57.490352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:57.490365 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:57.491256 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:53:57.491257 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:57.491285 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:53:57.491314 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:57.491523 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:57.491529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:57.491533 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:57.491632 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 22:53:57.491695 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260928 22:53:57.492399 19461 heartbeater.cc:347] Connected to a master server at 127.14.14.190:45925
I20260928 22:53:57.492507 19461 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:57.492657 19461 heartbeater.cc:511] Master 127.14.14.190:45925 requested a full tablet report, sending...
I20260928 22:53:57.493192 19011 ts_manager.cc:194] Registered new tserver with Master: 33cd5195f64941e6986c6afd3120c8db (127.14.14.131:43349)
I20260928 22:53:57.495594 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 19333
I20260928 22:53:57.495796 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:53:57.496071 14394 external_mini_cluster.cc:1428] Setting key 05a556219fc30a32846208548a1ea5d9
I20260928 22:53:57.497066 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:53:57.499614 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.498456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.14.14.190:45925)
0928 22:53:57.498531 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:57.498534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:57.498662 (+ 128us) server_negotiation.cc:407] Connection header received
0928 22:53:57.498701 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:57.498701 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:57.498706 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:57.498744 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:57.499415 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:57.499415 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:57.499416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:57.499418 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:57.499430 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:57.499433 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:57.499471 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:57.499504 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 22:53:57.499516 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 22:53:57.501094 19011 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:53:57.502614 19011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:53:57.506445 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.504703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:41843 (local address 127.0.0.1:38838)
0928 22:53:57.505137 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:57.505149 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:57.505199 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:57.505473 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:57.505474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:57.505477 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:57.506060 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:53:57.506061 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:57.506066 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:53:57.506077 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:57.506281 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:57.506286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:57.506289 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:53:57.506361 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 22:53:57.506374 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":71,"threads_started":1}
I20260928 22:53:57.506466 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.504981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.14.14.130:41843)
0928 22:53:57.505342 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:57.505346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:57.505355 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:53:57.505379 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:57.505380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:57.505387 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:57.505472 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:57.506155 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:57.506163 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:57.506164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:57.506168 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:57.506201 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:57.506226 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:57.506287 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:57.506375 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 22:53:57.506390 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":87,"threads_started":1}
I20260928 22:53:57.506791 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.504980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.14.14.131:43349)
0928 22:53:57.505846 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:57.505850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:57.505870 (+ 20us) server_negotiation.cc:407] Connection header received
0928 22:53:57.505895 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:57.505897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:57.505905 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:57.505983 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:57.506506 (+ 523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:57.506512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:57.506513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:57.506518 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:57.506552 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:57.506575 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:57.506633 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:57.506712 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 22:53:57.506732 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":97,"threads_started":1}
I20260928 22:53:57.506760 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.504698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:43349 (local address 127.0.0.1:53104)
0928 22:53:57.504819 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:57.504856 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:57.504935 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:57.506001 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:57.506002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:57.506005 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:57.506392 (+ 387us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:53:57.506392 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:57.506401 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:53:57.506415 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:57.506642 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:57.506643 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:57.506644 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 22:53:57.506684 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 22:53:57.506696 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":87,"threads_started":1}
I20260928 22:53:57.506994 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.504949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46016 (local address 127.14.14.129:36947)
0928 22:53:57.505124 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:57.505129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:57.505217 (+ 88us) server_negotiation.cc:407] Connection header received
0928 22:53:57.505280 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:57.505281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:57.505290 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:57.505384 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:57.506531 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:57.506534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:57.506534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:57.506537 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:57.506562 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:57.506575 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:57.506637 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:57.506879 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 22:53:57.506898 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260928 22:53:57.506903 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.504718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:36947 (local address 127.0.0.1:46016)
0928 22:53:57.505165 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:57.505174 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:57.505218 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:57.505385 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:57.505391 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:57.505398 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:57.505910 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:53:57.505910 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:57.505929 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:53:57.505951 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:57.506798 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:57.506799 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:57.506799 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 22:53:57.506843 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:53:57.506852 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":84,"threads_started":1}
I20260928 22:53:57.507579 19395 tablet_service.cc:1467] Processing CreateTablet for tablet 0a52e2996a6846fd8d36b22413a67b54 (DEFAULT_TABLE table=test-table [id=71bd3a058756444daef12eb958eb0cda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:57.507665 19133 tablet_service.cc:1467] Processing CreateTablet for tablet 0a52e2996a6846fd8d36b22413a67b54 (DEFAULT_TABLE table=test-table [id=71bd3a058756444daef12eb958eb0cda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:57.507925 19133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a52e2996a6846fd8d36b22413a67b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:57.507925 19395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a52e2996a6846fd8d36b22413a67b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:57.510092 19264 tablet_service.cc:1467] Processing CreateTablet for tablet 0a52e2996a6846fd8d36b22413a67b54 (DEFAULT_TABLE table=test-table [id=71bd3a058756444daef12eb958eb0cda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:57.510439 19264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a52e2996a6846fd8d36b22413a67b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:57.510658 19475 tablet_bootstrap.cc:492] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5: Bootstrap starting.
I20260928 22:53:57.511003 19476 tablet_bootstrap.cc:492] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db: Bootstrap starting.
I20260928 22:53:57.511361 19475 tablet_bootstrap.cc:654] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:57.511622 19476 tablet_bootstrap.cc:654] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:57.511664 19475 log.cc:824] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:57.511912 19476 log.cc:824] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:57.512442 19475 tablet_bootstrap.cc:492] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5: No bootstrap required, opened a new log
I20260928 22:53:57.512485 19475 ts_tablet_manager.cc:1397] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:53:57.512630 19476 tablet_bootstrap.cc:492] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db: No bootstrap required, opened a new log
I20260928 22:53:57.512687 19476 ts_tablet_manager.cc:1397] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:53:57.513417 19479 tablet_bootstrap.cc:492] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8: Bootstrap starting.
I20260928 22:53:57.514072 19479 tablet_bootstrap.cc:654] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:57.514430 19479 log.cc:824] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:57.514740 19475 raft_consensus.cc:348] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cd5195f64941e6986c6afd3120c8db" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 43349 } } peers { permanent_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 41843 } } peers { permanent_uuid: "5a4528c147d94db98946565c9abb93c5" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36947 } }
I20260928 22:53:57.514909 19475 raft_consensus.cc:374] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:57.514964 19475 raft_consensus.cc:729] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4528c147d94db98946565c9abb93c5, State: Initialized, Role: FOLLOWER
I20260928 22:53:57.515085 19475 consensus_queue.cc:260] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cd5195f64941e6986c6afd3120c8db" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 43349 } } peers { permanent_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 41843 } } peers { permanent_uuid: "5a4528c147d94db98946565c9abb93c5" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36947 } }
I20260928 22:53:57.515251 19479 tablet_bootstrap.cc:492] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8: No bootstrap required, opened a new log
I20260928 22:53:57.515350 19475 ts_tablet_manager.cc:1428] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:53:57.515252 19476 raft_consensus.cc:348] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cd5195f64941e6986c6afd3120c8db" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 43349 } } peers { permanent_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 41843 } } peers { permanent_uuid: "5a4528c147d94db98946565c9abb93c5" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36947 } }
I20260928 22:53:57.515376 19199 heartbeater.cc:503] Master 127.14.14.190:45925 was elected leader, sending a full tablet report...
I20260928 22:53:57.515409 19476 raft_consensus.cc:374] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:57.515458 19476 raft_consensus.cc:729] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33cd5195f64941e6986c6afd3120c8db, State: Initialized, Role: FOLLOWER
I20260928 22:53:57.515336 19479 ts_tablet_manager.cc:1397] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:53:57.515571 19476 consensus_queue.cc:260] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cd5195f64941e6986c6afd3120c8db" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 43349 } } peers { permanent_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 41843 } } peers { permanent_uuid: "5a4528c147d94db98946565c9abb93c5" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36947 } }
I20260928 22:53:57.515868 19461 heartbeater.cc:503] Master 127.14.14.190:45925 was elected leader, sending a full tablet report...
I20260928 22:53:57.515859 19476 ts_tablet_manager.cc:1428] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:53:57.517956 19479 raft_consensus.cc:348] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cd5195f64941e6986c6afd3120c8db" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 43349 } } peers { permanent_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 41843 } } peers { permanent_uuid: "5a4528c147d94db98946565c9abb93c5" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36947 } }
I20260928 22:53:57.518072 19479 raft_consensus.cc:374] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:57.518093 19479 raft_consensus.cc:729] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e2d583d7a9b4a1191d21904d0969ea8, State: Initialized, Role: FOLLOWER
I20260928 22:53:57.518204 19479 consensus_queue.cc:260] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cd5195f64941e6986c6afd3120c8db" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 43349 } } peers { permanent_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 41843 } } peers { permanent_uuid: "5a4528c147d94db98946565c9abb93c5" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36947 } }
I20260928 22:53:57.518451 19479 ts_tablet_manager.cc:1428] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:53:57.518491 19330 heartbeater.cc:503] Master 127.14.14.190:45925 was elected leader, sending a full tablet report...
W20260928 22:53:57.620122 19331 tablet.cc:2369] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:53:57.733265 19483 raft_consensus.cc:482] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:53:57.733408 19483 raft_consensus.cc:504] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cd5195f64941e6986c6afd3120c8db" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 43349 } } peers { permanent_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 41843 } } peers { permanent_uuid: "5a4528c147d94db98946565c9abb93c5" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36947 } }
I20260928 22:53:57.733927 19483 leader_election.cc:288] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33cd5195f64941e6986c6afd3120c8db (127.14.14.131:43349), 5a4528c147d94db98946565c9abb93c5 (127.14.14.129:36947)
I20260928 22:53:57.734130 19482 raft_consensus.cc:482] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:53:57.734270 19482 raft_consensus.cc:504] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cd5195f64941e6986c6afd3120c8db" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 43349 } } peers { permanent_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 41843 } } peers { permanent_uuid: "5a4528c147d94db98946565c9abb93c5" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36947 } }
I20260928 22:53:57.734977 19482 leader_election.cc:288] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e2d583d7a9b4a1191d21904d0969ea8 (127.14.14.130:41843), 5a4528c147d94db98946565c9abb93c5 (127.14.14.129:36947)
I20260928 22:53:57.735390 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.734123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:43349 (local address 127.14.14.130:55127)
0928 22:53:57.734180 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:57.734192 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:57.734235 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:57.734443 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:57.734444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:57.734446 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:57.734950 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:53:57.734950 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:57.734958 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:53:57.734972 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:57.735265 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:57.735267 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:57.735267 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 22:53:57.735322 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:53:57.735342 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:53:57.735713 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.734262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:55127 (local address 127.14.14.131:43349)
0928 22:53:57.734307 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:57.734309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:57.734320 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:53:57.734350 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:57.734352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:57.734357 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:57.734427 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:57.735148 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:57.735150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:57.735150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:57.735154 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:57.735170 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:57.735175 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:57.735235 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:57.735624 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 22:53:57.735638 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:53:57.735852 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.734132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:36947 (local address 127.14.14.130:44643)
0928 22:53:57.734341 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:57.734349 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:57.734407 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:57.734783 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:57.734784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:57.734787 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:57.735525 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:53:57.735526 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:57.735531 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:53:57.735543 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:57.735700 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:57.735701 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:57.735701 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 22:53:57.735755 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 22:53:57.735768 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
I20260928 22:53:57.735891 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.734299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:44643 (local address 127.14.14.129:36947)
0928 22:53:57.734350 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:57.734352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:57.734612 (+ 260us) server_negotiation.cc:407] Connection header received
0928 22:53:57.734667 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:57.734668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:57.734675 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:57.734727 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:57.735603 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:57.735604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:57.735605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:57.735609 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:57.735623 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:57.735628 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:57.735677 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:57.735786 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 22:53:57.735804 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:53:57.736039 19415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a52e2996a6846fd8d36b22413a67b54" candidate_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33cd5195f64941e6986c6afd3120c8db" is_pre_election: true
I20260928 22:53:57.736136 19415 raft_consensus.cc:2454] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e2d583d7a9b4a1191d21904d0969ea8 in term 0.
I20260928 22:53:57.736178 19153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a52e2996a6846fd8d36b22413a67b54" candidate_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4528c147d94db98946565c9abb93c5" is_pre_election: true
I20260928 22:53:57.736311 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.734766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:36947 (local address 127.14.14.131:38361)
0928 22:53:57.734979 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:57.734989 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:57.735032 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:57.735336 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:57.735337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:57.735340 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:57.735964 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:53:57.735965 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:57.735972 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:53:57.735987 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:57.736189 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:57.736189 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:57.736190 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 22:53:57.736233 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 22:53:57.736245 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260928 22:53:57.736320 19153 raft_consensus.cc:2454] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e2d583d7a9b4a1191d21904d0969ea8 in term 0.
I20260928 22:53:57.736445 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.735269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:55531 (local address 127.14.14.130:41843)
0928 22:53:57.735329 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:57.735331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:57.735338 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:53:57.735356 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:57.735357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:57.735362 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:57.735422 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:57.736131 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:57.736132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:57.736133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:57.736137 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:57.736149 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:57.736153 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:57.736199 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:57.736334 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 22:53:57.736347 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:53:57.736480 19218 leader_election.cc:302] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33cd5195f64941e6986c6afd3120c8db, 6e2d583d7a9b4a1191d21904d0969ea8; no voters:
I20260928 22:53:57.736515 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.735036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:38361 (local address 127.14.14.129:36947)
0928 22:53:57.735152 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:57.735153 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:57.735160 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:53:57.735186 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:57.735187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:57.735191 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:57.735263 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:57.736072 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:57.736073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:57.736073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:57.736077 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:57.736115 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:57.736120 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:57.736164 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:57.736414 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 22:53:57.736428 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":80,"threads_started":1}
I20260928 22:53:57.736572 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.734749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:41843 (local address 127.14.14.131:55531)
0928 22:53:57.734851 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:57.734888 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:57.734924 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:57.735446 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:57.735447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:57.735449 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:57.736035 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:53:57.736036 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:57.736042 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:53:57.736053 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:57.736222 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:57.736223 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:57.736223 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 22:53:57.736270 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 22:53:57.736283 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 22:53:57.736642 19483 raft_consensus.cc:2781] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:53:57.736696 19483 raft_consensus.cc:482] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:53:57.736752 19483 raft_consensus.cc:3037] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:57.736773 19153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a52e2996a6846fd8d36b22413a67b54" candidate_uuid: "33cd5195f64941e6986c6afd3120c8db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4528c147d94db98946565c9abb93c5" is_pre_election: true
I20260928 22:53:57.736840 19153 raft_consensus.cc:2454] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33cd5195f64941e6986c6afd3120c8db in term 0.
I20260928 22:53:57.736920 19284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a52e2996a6846fd8d36b22413a67b54" candidate_uuid: "33cd5195f64941e6986c6afd3120c8db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" is_pre_election: true
I20260928 22:53:57.737030 19348 leader_election.cc:302] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33cd5195f64941e6986c6afd3120c8db, 5a4528c147d94db98946565c9abb93c5; no voters:
I20260928 22:53:57.737232 19482 raft_consensus.cc:2781] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:53:57.737337 19482 raft_consensus.cc:482] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:53:57.737360 19482 raft_consensus.cc:3037] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:57.737692 19483 raft_consensus.cc:504] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cd5195f64941e6986c6afd3120c8db" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 43349 } } peers { permanent_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 41843 } } peers { permanent_uuid: "5a4528c147d94db98946565c9abb93c5" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36947 } }
I20260928 22:53:57.737857 19284 raft_consensus.cc:2379] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 33cd5195f64941e6986c6afd3120c8db in current term 1: Already voted for candidate 6e2d583d7a9b4a1191d21904d0969ea8 in this term.
I20260928 22:53:57.737874 19483 leader_election.cc:288] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [CANDIDATE]: Term 1 election: Requested vote from peers 33cd5195f64941e6986c6afd3120c8db (127.14.14.131:43349), 5a4528c147d94db98946565c9abb93c5 (127.14.14.129:36947)
I20260928 22:53:57.738054 19415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a52e2996a6846fd8d36b22413a67b54" candidate_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33cd5195f64941e6986c6afd3120c8db"
I20260928 22:53:57.738063 19153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a52e2996a6846fd8d36b22413a67b54" candidate_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4528c147d94db98946565c9abb93c5"
I20260928 22:53:57.738135 19153 raft_consensus.cc:3037] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:57.738276 19482 raft_consensus.cc:504] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cd5195f64941e6986c6afd3120c8db" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 43349 } } peers { permanent_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 41843 } } peers { permanent_uuid: "5a4528c147d94db98946565c9abb93c5" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36947 } }
I20260928 22:53:57.738364 19415 raft_consensus.cc:2379] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e2d583d7a9b4a1191d21904d0969ea8 in current term 1: Already voted for candidate 33cd5195f64941e6986c6afd3120c8db in this term.
I20260928 22:53:57.738511 19482 leader_election.cc:288] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [CANDIDATE]: Term 1 election: Requested vote from peers 6e2d583d7a9b4a1191d21904d0969ea8 (127.14.14.130:41843), 5a4528c147d94db98946565c9abb93c5 (127.14.14.129:36947)
I20260928 22:53:57.738648 19284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a52e2996a6846fd8d36b22413a67b54" candidate_uuid: "33cd5195f64941e6986c6afd3120c8db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e2d583d7a9b4a1191d21904d0969ea8"
I20260928 22:53:57.738727 19284 raft_consensus.cc:2379] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33cd5195f64941e6986c6afd3120c8db in current term 1: Already voted for candidate 6e2d583d7a9b4a1191d21904d0969ea8 in this term.
I20260928 22:53:57.738687 19152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a52e2996a6846fd8d36b22413a67b54" candidate_uuid: "33cd5195f64941e6986c6afd3120c8db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4528c147d94db98946565c9abb93c5"
I20260928 22:53:57.738994 19348 leader_election.cc:302] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33cd5195f64941e6986c6afd3120c8db; no voters: 5a4528c147d94db98946565c9abb93c5, 6e2d583d7a9b4a1191d21904d0969ea8
I20260928 22:53:57.739070 19153 raft_consensus.cc:2454] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e2d583d7a9b4a1191d21904d0969ea8 in term 1.
W20260928 22:53:57.739038 19462 tablet.cc:2369] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:53:57.739116 19482 raft_consensus.cc:2726] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 22:53:57.739233 19217 leader_election.cc:302] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4528c147d94db98946565c9abb93c5, 6e2d583d7a9b4a1191d21904d0969ea8; no voters: 33cd5195f64941e6986c6afd3120c8db
I20260928 22:53:57.739321 19483 raft_consensus.cc:2781] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:57.739459 19483 raft_consensus.cc:686] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [term 1 LEADER]: Becoming Leader. State: Replica: 6e2d583d7a9b4a1191d21904d0969ea8, State: Running, Role: LEADER
I20260928 22:53:57.739603 19483 consensus_queue.cc:237] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cd5195f64941e6986c6afd3120c8db" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 43349 } } peers { permanent_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 41843 } } peers { permanent_uuid: "5a4528c147d94db98946565c9abb93c5" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36947 } }
I20260928 22:53:57.740389 19011 catalog_manager.cc:5495] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e2d583d7a9b4a1191d21904d0969ea8 (127.14.14.130). New cstate: current_term: 1 leader_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cd5195f64941e6986c6afd3120c8db" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 43349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e2d583d7a9b4a1191d21904d0969ea8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 41843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4528c147d94db98946565c9abb93c5" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36947 } health_report { overall_health: UNKNOWN } } }
W20260928 22:53:57.746282 19200 tablet.cc:2369] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:53:57.799039 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.797917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38852 (local address 127.14.14.130:41843)
0928 22:53:57.797982 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:57.797984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:57.798000 (+ 16us) server_negotiation.cc:407] Connection header received
0928 22:53:57.798038 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:57.798039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:57.798054 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:57.798120 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:57.798818 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:57.798819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:57.798819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:57.798823 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:57.798843 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:57.798847 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:57.798888 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:57.798974 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 22:53:57.798985 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:53:57.800879 19153 raft_consensus.cc:1261] T 0a52e2996a6846fd8d36b22413a67b54 P 5a4528c147d94db98946565c9abb93c5 [term 1 FOLLOWER]: Refusing update from remote peer 6e2d583d7a9b4a1191d21904d0969ea8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:57.800958 19415 raft_consensus.cc:1261] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [term 1 FOLLOWER]: Refusing update from remote peer 6e2d583d7a9b4a1191d21904d0969ea8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:57.801205 19487 consensus_queue.cc:1035] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4528c147d94db98946565c9abb93c5" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 36947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:57.801452 19483 consensus_queue.cc:1035] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33cd5195f64941e6986c6afd3120c8db" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 43349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:57.804639 19492 mvcc.cc:205] Tried to move back new op lower bound from 7334445210829762560 to 7334445210581929984. Current Snapshot: MvccSnapshot[applied={T|T < 7334445210829762560}]
I20260928 22:53:57.807363 19011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:57.807456 19011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:57.808195 19011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe1f6d6eece14cddb72d1da5f2ce0d98: Sending DeleteTablet for 3 replicas of tablet 0a52e2996a6846fd8d36b22413a67b54
I20260928 22:53:57.808523 19264 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a52e2996a6846fd8d36b22413a67b54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:57 UTC) from {username='slave'} at 127.0.0.1:38838
I20260928 22:53:57.808547 19395 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a52e2996a6846fd8d36b22413a67b54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:57 UTC) from {username='slave'} at 127.0.0.1:53104
I20260928 22:53:57.808770 19499 tablet_replica.cc:330] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8: stopping tablet replica
I20260928 22:53:57.808971 19500 tablet_replica.cc:330] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db: stopping tablet replica
I20260928 22:53:57.808933 19499 raft_consensus.cc:2229] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:53:57.809062 19499 raft_consensus.cc:2258] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:57.809128 19500 raft_consensus.cc:2229] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:57.809131 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 19071
I20260928 22:53:57.809536 19499 ts_tablet_manager.cc:1905] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:57.810259 19500 raft_consensus.cc:2258] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:57.811650 19499 ts_tablet_manager.cc:1918] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:53:57.811762 19499 log.cc:1196] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/0a52e2996a6846fd8d36b22413a67b54
I20260928 22:53:57.812074 19499 ts_tablet_manager.cc:1939] T 0a52e2996a6846fd8d36b22413a67b54 P 6e2d583d7a9b4a1191d21904d0969ea8: Deleting consensus metadata
I20260928 22:53:57.812485 18998 catalog_manager.cc:4841] TS 6e2d583d7a9b4a1191d21904d0969ea8 (127.14.14.130:41843): tablet 0a52e2996a6846fd8d36b22413a67b54 (table test-table [id=71bd3a058756444daef12eb958eb0cda]) successfully deleted
I20260928 22:53:57.814972 19500 ts_tablet_manager.cc:1905] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:57.816425 19500 ts_tablet_manager.cc:1918] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:53:57.816514 19500 log.cc:1196] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/0a52e2996a6846fd8d36b22413a67b54
I20260928 22:53:57.816792 19500 ts_tablet_manager.cc:1939] T 0a52e2996a6846fd8d36b22413a67b54 P 33cd5195f64941e6986c6afd3120c8db: Deleting consensus metadata
W20260928 22:53:57.817202 18995 connection.cc:535] client connection to 127.14.14.129:36947 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 22:53:57.817317 18995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 22:53:57.817422 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 19202
I20260928 22:53:57.817593 18996 catalog_manager.cc:4841] TS 33cd5195f64941e6986c6afd3120c8db (127.14.14.131:43349): tablet 0a52e2996a6846fd8d36b22413a67b54 (table test-table [id=71bd3a058756444daef12eb958eb0cda]) successfully deleted
I20260928 22:53:57.818074 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.817834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:36947 (local address 127.0.0.1:46018)
0928 22:53:57.817987 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:57.818039 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260928 22:53:57.818235 18995 catalog_manager.cc:4584] TS 5a4528c147d94db98946565c9abb93c5 (127.14.14.129:36947): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a52e2996a6846fd8d36b22413a67b54: Network error: Client connection negotiation failed: client connection to 127.14.14.129:36947: connect: Connection refused (error 111)
I20260928 22:53:57.823212 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 19333
I20260928 22:53:57.827540 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 18980
2026-09-28T22:53:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (857 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 22:53:57.847034 14394 mini_oidc.cc:152] Starting JWKS server
I20260928 22:53:57.848122 14394 webserver.cc:460] Webserver started at https://127.0.0.1:45865/ using document root <none> and password file <none>
I20260928 22:53:57.848249 14394 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 22:53:57.848436 14394 webserver.cc:460] Webserver started at http://127.0.0.1:40235/ using document root <none> and password file <none>
2026-09-28T22:53:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:57Z Disabled control of system clock
I20260928 22:53:57.855923 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:35201
--webserver_interface=127.14.14.190
--webserver_port=0
--builtin_ntp_servers=127.14.14.148:40523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:35201
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45865/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/testchainca.cert with env {}
W20260928 22:53:57.937251 19507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:57.937470 19507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:57.937494 19507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:57.938773 19507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:57.938822 19507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:57.938835 19507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:57.938848 19507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:57.938898 19507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 22:53:57.938931 19507 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 22:53:57.938947 19507 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45865/jwks.json
I20260928 22:53:57.940532 19507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:35201
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:35201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45865/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:57.940834 19507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:57.941071 19507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:57.946131 19515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:57.946132 19512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:57.946137 19513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:57.946436 19507 server_base.cc:1034] running on GCE node
I20260928 22:53:57.946712 19507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:57.947026 19507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:57.948196 19507 hybrid_clock.cc:648] HybridClock initialized: now 1790636037948184 us; error 33 us; skew 500 ppm
I20260928 22:53:57.949424 19507 webserver.cc:460] Webserver started at http://127.14.14.190:40147/ using document root <none> and password file <none>
I20260928 22:53:57.949725 19507 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:57.949775 19507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:57.949873 19507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:57.950922 19507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "71419218c41946408fb4ece53b90b97f"
format_stamp: "Formatted at 2026-09-28 22:53:57 on dist-test-slave-kzfr"
server_key: "858cc6a0c93dbb2bbfc1b60242c115c7"
server_key_iv: "fe28edff82ce0769fa33e642044d09ef"
server_key_version: "encryptionkey@0"
I20260928 22:53:57.951226 19507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "71419218c41946408fb4ece53b90b97f"
format_stamp: "Formatted at 2026-09-28 22:53:57 on dist-test-slave-kzfr"
server_key: "858cc6a0c93dbb2bbfc1b60242c115c7"
server_key_iv: "fe28edff82ce0769fa33e642044d09ef"
server_key_version: "encryptionkey@0"
I20260928 22:53:57.952440 19507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:53:57.953083 19521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:57.953244 19507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:53:57.953310 19507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "71419218c41946408fb4ece53b90b97f"
format_stamp: "Formatted at 2026-09-28 22:53:57 on dist-test-slave-kzfr"
server_key: "858cc6a0c93dbb2bbfc1b60242c115c7"
server_key_iv: "fe28edff82ce0769fa33e642044d09ef"
server_key_version: "encryptionkey@0"
I20260928 22:53:57.953362 19507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:57.971122 19507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:57.971413 19507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:57.971519 19507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:57.975688 19507 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:35201
I20260928 22:53:57.975725 19573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:35201 every 8 connection(s)
I20260928 22:53:57.976099 19507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:57.977059 19574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:57.979363 19574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f: Bootstrap starting.
I20260928 22:53:57.979915 19574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:57.980176 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 19507
I20260928 22:53:57.980288 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:53:57.980268 19574 log.cc:824] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:57.980494 14394 external_mini_cluster.cc:1428] Setting key afa6ec8ae317910195eb9c2868eb3fed
I20260928 22:53:57.981048 19574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f: No bootstrap required, opened a new log
I20260928 22:53:57.982895 19574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71419218c41946408fb4ece53b90b97f" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35201 } }
I20260928 22:53:57.983031 19574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:57.983053 19574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71419218c41946408fb4ece53b90b97f, State: Initialized, Role: FOLLOWER
I20260928 22:53:57.983192 19574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71419218c41946408fb4ece53b90b97f" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35201 } }
I20260928 22:53:57.983268 19574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:57.983290 19574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:57.983367 19574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:57.983924 19574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71419218c41946408fb4ece53b90b97f" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35201 } }
I20260928 22:53:57.984038 19574 leader_election.cc:302] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71419218c41946408fb4ece53b90b97f; no voters:
I20260928 22:53:57.984242 19574 leader_election.cc:288] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:57.984273 19579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:57.984387 19579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [term 1 LEADER]: Becoming Leader. State: Replica: 71419218c41946408fb4ece53b90b97f, State: Running, Role: LEADER
I20260928 22:53:57.984469 19579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71419218c41946408fb4ece53b90b97f" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35201 } }
I20260928 22:53:57.984510 19574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:57.984776 19579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71419218c41946408fb4ece53b90b97f. Latest consensus state: current_term: 1 leader_uuid: "71419218c41946408fb4ece53b90b97f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71419218c41946408fb4ece53b90b97f" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35201 } } }
I20260928 22:53:57.984788 19581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71419218c41946408fb4ece53b90b97f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71419218c41946408fb4ece53b90b97f" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35201 } } }
I20260928 22:53:57.985021 19579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:57.985030 19581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:57.985291 19588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:57.985829 19588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:57.987524 19588 catalog_manager.cc:1347] Generated new cluster ID: b8f919b7c7f64a74b76d609a851c1057
I20260928 22:53:57.987572 19588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:57.987942 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:57.981077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.14.14.190:35201)
0928 22:53:57.981290 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:57.981298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:57.981322 (+ 24us) server_negotiation.cc:407] Connection header received
0928 22:53:57.982250 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:57.982258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:57.982322 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:57.982491 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:57.982701 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:57.984081 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:57.985207 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:57.985993 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:57.986094 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:57.987238 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:57.987241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:57.987241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:57.987465 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:57.987508 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:57.987523 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:57.987606 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:57.987703 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 22:53:57.987784 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260928 22:53:57.989048 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:58.003847 19588 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:58.004393 19588 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:58.007678 19588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71419218c41946408fb4ece53b90b97f: Generated new TSK 0
I20260928 22:53:58.007887 19588 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 22:53:58.037863 19604 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 22:53:58.037983 19604 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 22:53:58.038501 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:58.015742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60296 (local address 127.14.14.190:35201)
0928 22:53:58.015786 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:58.015788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:58.015901 (+ 113us) server_negotiation.cc:407] Connection header received
0928 22:53:58.016000 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:58.016000 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:58.016009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:58.016061 (+ 52us) server_negotiation.cc:226] Negotiated authn=JWT
0928 22:53:58.016194 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.016674 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.017798 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.017998 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.018052 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:58.018283 (+ 231us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 22:53:58.038160 (+ 19877us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 22:53:58.038448 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60296: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 22:53:58.038602 19578 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60296: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 22:53:58.038918 19603 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:58.015619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35201 (local address 127.0.0.1:60296)
0928 22:53:58.015814 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:58.015826 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:58.015911 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:58.016076 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:58.016077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:58.016079 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 22:53:58.016141 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:58.016143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.016748 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.016749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:58.017721 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:58.017724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.018084 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.018085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:58.018153 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:58.018175 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 22:53:58.038438 (+ 20263us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 22:53:58.038568 (+ 130us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.14.190:35201: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.14.190:35201: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 22:53:58.039443 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 19507
2026-09-28T22:53:58Z chronyd exiting
I20260928 22:53:58.075018 14394 test_util.cc:175] -----------------------------------------------
I20260928 22:53:58.075096 14394 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790636016256058-14394-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (229 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 22:53:58.075803 14394 mini_oidc.cc:152] Starting JWKS server
I20260928 22:53:58.076452 14394 webserver.cc:460] Webserver started at https://127.0.0.1:33069/ using document root <none> and password file <none>
I20260928 22:53:58.076491 14394 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 22:53:58.076632 14394 webserver.cc:460] Webserver started at http://127.0.0.1:33939/ using document root <none> and password file <none>
2026-09-28T22:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:58Z Disabled control of system clock
I20260928 22:53:58.083871 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:34261
--webserver_interface=127.14.14.190
--webserver_port=0
--builtin_ntp_servers=127.14.14.148:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:34261
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33069/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/testchainca.cert with env {}
W20260928 22:53:58.162796 19610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:58.163048 19610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:58.163087 19610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:58.164258 19610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:58.164330 19610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:58.164351 19610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:58.164382 19610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:58.164428 19610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 22:53:58.164461 19610 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 22:53:58.164489 19610 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33069/jwks.json
I20260928 22:53:58.166054 19610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:34261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:34261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33069/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:58.166318 19610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:58.166543 19610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:58.171528 19616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:58.171550 19618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:58.171541 19615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:58.171733 19610 server_base.cc:1034] running on GCE node
I20260928 22:53:58.171947 19610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:58.172190 19610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:58.173327 19610 hybrid_clock.cc:648] HybridClock initialized: now 1790636038173313 us; error 28 us; skew 500 ppm
I20260928 22:53:58.174568 19610 webserver.cc:460] Webserver started at http://127.14.14.190:44281/ using document root <none> and password file <none>
I20260928 22:53:58.174813 19610 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:58.174858 19610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:58.175022 19610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:58.176110 19610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "90a653b783344c6ba69ff3288970a7fe"
format_stamp: "Formatted at 2026-09-28 22:53:58 on dist-test-slave-kzfr"
server_key: "13ff44d969e0be650523ab522986384b"
server_key_iv: "d27c0b0f8dd832c288310840785ead68"
server_key_version: "encryptionkey@0"
I20260928 22:53:58.176481 19610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "90a653b783344c6ba69ff3288970a7fe"
format_stamp: "Formatted at 2026-09-28 22:53:58 on dist-test-slave-kzfr"
server_key: "13ff44d969e0be650523ab522986384b"
server_key_iv: "d27c0b0f8dd832c288310840785ead68"
server_key_version: "encryptionkey@0"
I20260928 22:53:58.178193 19610 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260928 22:53:58.178903 19624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:58.179069 19610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:58.179173 19610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "90a653b783344c6ba69ff3288970a7fe"
format_stamp: "Formatted at 2026-09-28 22:53:58 on dist-test-slave-kzfr"
server_key: "13ff44d969e0be650523ab522986384b"
server_key_iv: "d27c0b0f8dd832c288310840785ead68"
server_key_version: "encryptionkey@0"
I20260928 22:53:58.179251 19610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:58.202191 19610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:58.202533 19610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:58.202700 19610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:58.206732 19610 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:34261
I20260928 22:53:58.206772 19676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:34261 every 8 connection(s)
I20260928 22:53:58.207119 19610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:58.207890 19677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:58.207966 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 19610
I20260928 22:53:58.208076 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:53:58.208276 14394 external_mini_cluster.cc:1428] Setting key 39d56ef343ca944f2f09817803ac1261
I20260928 22:53:58.211313 19677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe: Bootstrap starting.
I20260928 22:53:58.211902 19677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:58.212198 19677 log.cc:824] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:58.212936 19677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe: No bootstrap required, opened a new log
I20260928 22:53:58.213809 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:58.208893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44876 (local address 127.14.14.190:34261)
0928 22:53:58.209115 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:58.209123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:58.209147 (+ 24us) server_negotiation.cc:407] Connection header received
0928 22:53:58.209639 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:58.209652 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:58.209714 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:58.209859 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:58.210048 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.210914 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.211775 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.212240 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.212398 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:58.213082 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:58.213086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:58.213086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:58.213257 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:58.213291 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:58.213301 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:58.213358 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:58.213472 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 22:53:58.213567 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":101,"threads_started":1}
I20260928 22:53:58.214980 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:58.215183 19677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a653b783344c6ba69ff3288970a7fe" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 34261 } }
I20260928 22:53:58.215315 19677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:58.215338 19677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a653b783344c6ba69ff3288970a7fe, State: Initialized, Role: FOLLOWER
I20260928 22:53:58.215476 19677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a653b783344c6ba69ff3288970a7fe" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 34261 } }
I20260928 22:53:58.215548 19677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:58.215571 19677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:58.215634 19677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:58.216171 19677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a653b783344c6ba69ff3288970a7fe" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 34261 } }
I20260928 22:53:58.216287 19677 leader_election.cc:302] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90a653b783344c6ba69ff3288970a7fe; no voters:
I20260928 22:53:58.216459 19677 leader_election.cc:288] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:58.216511 19682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:58.216676 19682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [term 1 LEADER]: Becoming Leader. State: Replica: 90a653b783344c6ba69ff3288970a7fe, State: Running, Role: LEADER
I20260928 22:53:58.216691 19677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:58.216768 19682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a653b783344c6ba69ff3288970a7fe" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 34261 } }
I20260928 22:53:58.217162 19683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90a653b783344c6ba69ff3288970a7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a653b783344c6ba69ff3288970a7fe" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 34261 } } }
I20260928 22:53:58.217171 19684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90a653b783344c6ba69ff3288970a7fe. Latest consensus state: current_term: 1 leader_uuid: "90a653b783344c6ba69ff3288970a7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a653b783344c6ba69ff3288970a7fe" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 34261 } } }
I20260928 22:53:58.217285 19683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:58.217312 19684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:58.217573 19691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:58.218123 19691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:58.219709 19691 catalog_manager.cc:1347] Generated new cluster ID: 6a67cd21bbc249c1b6a83e64c05abb84
I20260928 22:53:58.219776 19691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:58.226496 19691 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:58.226994 19691 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:58.230087 19691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90a653b783344c6ba69ff3288970a7fe: Generated new TSK 0
I20260928 22:53:58.230247 19691 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 22:53:58.252077 19707 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 22:53:58.252180 19707 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 22:53:58.252486 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:58.236847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44886 (local address 127.14.14.190:34261)
0928 22:53:58.236899 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:58.236901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:58.237001 (+ 100us) server_negotiation.cc:407] Connection header received
0928 22:53:58.237042 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:58.237043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:58.237056 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:58.237133 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0928 22:53:58.237253 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.238065 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.238814 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.239160 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.239238 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:58.239378 (+ 140us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 22:53:58.252255 (+ 12877us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 22:53:58.252414 (+ 159us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 22:53:58.252547 19680 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 22:53:58.252621 19706 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:58.236775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:34261 (local address 127.0.0.1:44886)
0928 22:53:58.236924 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:58.236934 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:58.236995 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:58.237138 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:58.237139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:58.237140 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 22:53:58.237192 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:58.237194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.238165 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.238165 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:58.238740 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:58.238741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.239245 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.239246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:58.239285 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:58.239289 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 22:53:58.252386 (+ 13097us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 22:53:58.252502 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.14.190:34261: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260928 22:53:58.253048 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 19610
2026-09-28T22:53:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 22:53:58.290460 14394 mini_oidc.cc:152] Starting JWKS server
I20260928 22:53:58.291158 14394 webserver.cc:460] Webserver started at https://127.0.0.1:43605/ using document root <none> and password file <none>
I20260928 22:53:58.291227 14394 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 22:53:58.291394 14394 webserver.cc:460] Webserver started at http://127.0.0.1:42775/ using document root <none> and password file <none>
2026-09-28T22:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:58Z Disabled control of system clock
I20260928 22:53:58.300014 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:43709
--webserver_interface=127.14.14.190
--webserver_port=0
--builtin_ntp_servers=127.14.14.148:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:43709
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43605/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 22:53:58.382601 19713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:58.382812 19713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:58.382841 19713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:58.384078 19713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:58.384120 19713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:58.384135 19713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:58.384150 19713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:58.384166 19713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 22:53:58.384178 19713 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 22:53:58.384191 19713 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43605/jwks.json
I20260928 22:53:58.385525 19713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:43709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:43709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43605/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:58.385833 19713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:58.386106 19713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:58.391170 19721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:58.391177 19718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:58.391181 19719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:58.391376 19713 server_base.cc:1034] running on GCE node
I20260928 22:53:58.391695 19713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:58.392009 19713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:58.393173 19713 hybrid_clock.cc:648] HybridClock initialized: now 1790636038393149 us; error 39 us; skew 500 ppm
I20260928 22:53:58.398005 19713 webserver.cc:460] Webserver started at http://127.14.14.190:37667/ using document root <none> and password file <none>
I20260928 22:53:58.398257 19713 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:58.398334 19713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:58.398497 19713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:58.399598 19713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "c5090f9a506241b29f6ae2d2b31fdfe7"
format_stamp: "Formatted at 2026-09-28 22:53:58 on dist-test-slave-kzfr"
server_key: "2f421cc909f9e2bf052b9d662cdf99f0"
server_key_iv: "624b64125514a41ae591f867f1a3639f"
server_key_version: "encryptionkey@0"
I20260928 22:53:58.399973 19713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "c5090f9a506241b29f6ae2d2b31fdfe7"
format_stamp: "Formatted at 2026-09-28 22:53:58 on dist-test-slave-kzfr"
server_key: "2f421cc909f9e2bf052b9d662cdf99f0"
server_key_iv: "624b64125514a41ae591f867f1a3639f"
server_key_version: "encryptionkey@0"
I20260928 22:53:58.401310 19713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:58.402122 19727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:58.402326 19713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:58.402429 19713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "c5090f9a506241b29f6ae2d2b31fdfe7"
format_stamp: "Formatted at 2026-09-28 22:53:58 on dist-test-slave-kzfr"
server_key: "2f421cc909f9e2bf052b9d662cdf99f0"
server_key_iv: "624b64125514a41ae591f867f1a3639f"
server_key_version: "encryptionkey@0"
I20260928 22:53:58.402484 19713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:58.411826 19713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:58.412084 19713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:58.412184 19713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:58.416010 19713 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:43709
I20260928 22:53:58.416042 19779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:43709 every 8 connection(s)
I20260928 22:53:58.416391 19713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:58.417078 19780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:58.419848 19780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7: Bootstrap starting.
I20260928 22:53:58.420441 19780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:58.420737 19780 log.cc:824] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:58.421408 19780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7: No bootstrap required, opened a new log
I20260928 22:53:58.423041 19780 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5090f9a506241b29f6ae2d2b31fdfe7" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 43709 } }
I20260928 22:53:58.423202 19780 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:58.423252 19780 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5090f9a506241b29f6ae2d2b31fdfe7, State: Initialized, Role: FOLLOWER
I20260928 22:53:58.423424 19780 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5090f9a506241b29f6ae2d2b31fdfe7" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 43709 } }
I20260928 22:53:58.423524 19780 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:58.423569 19780 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:58.423620 19780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:58.424186 19780 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5090f9a506241b29f6ae2d2b31fdfe7" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 43709 } }
I20260928 22:53:58.424285 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 19713
I20260928 22:53:58.424306 19780 leader_election.cc:302] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5090f9a506241b29f6ae2d2b31fdfe7; no voters:
I20260928 22:53:58.424399 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:53:58.424469 19780 leader_election.cc:288] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:58.424517 19783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:58.424606 14394 external_mini_cluster.cc:1428] Setting key 056836e323d3c8952f01b74c06f5b3da
I20260928 22:53:58.424662 19783 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [term 1 LEADER]: Becoming Leader. State: Replica: c5090f9a506241b29f6ae2d2b31fdfe7, State: Running, Role: LEADER
I20260928 22:53:58.424721 19780 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:58.424763 19783 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5090f9a506241b29f6ae2d2b31fdfe7" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 43709 } }
I20260928 22:53:58.425051 19783 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5090f9a506241b29f6ae2d2b31fdfe7. Latest consensus state: current_term: 1 leader_uuid: "c5090f9a506241b29f6ae2d2b31fdfe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5090f9a506241b29f6ae2d2b31fdfe7" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 43709 } } }
I20260928 22:53:58.425137 19783 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:58.425056 19785 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5090f9a506241b29f6ae2d2b31fdfe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5090f9a506241b29f6ae2d2b31fdfe7" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 43709 } } }
I20260928 22:53:58.425256 19785 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:58.425833 19789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:58.426362 19789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:58.427935 19789 catalog_manager.cc:1347] Generated new cluster ID: 6423bacf578a4ff19af3fbd1ab16b9e1
I20260928 22:53:58.427989 19789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:58.430951 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:58.425284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.14.14.190:43709)
0928 22:53:58.426074 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:58.426083 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:58.426118 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:53:58.426240 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:58.426249 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:58.426320 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:58.426871 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:58.426983 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.428215 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.429075 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.429494 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.429661 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:58.430343 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:58.430347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:58.430347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:58.430525 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:58.430556 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:58.430566 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:58.430643 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:58.430716 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 22:53:58.430791 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":71,"threads_started":1}
I20260928 22:53:58.431802 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:58.449942 19789 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:58.450711 19789 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:58.456406 19789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5090f9a506241b29f6ae2d2b31fdfe7: Generated new TSK 0
I20260928 22:53:58.456629 19789 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:58.466789 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:58.463074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43752 (local address 127.14.14.190:43709)
0928 22:53:58.463117 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:58.463119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:58.463222 (+ 103us) server_negotiation.cc:407] Connection header received
0928 22:53:58.463293 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:58.463294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:58.463306 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:58.463382 (+ 76us) server_negotiation.cc:226] Negotiated authn=JWT
0928 22:53:58.463495 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.464296 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.465053 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.465395 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.465469 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:58.465641 (+ 172us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 22:53:58.466586 (+ 945us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43605/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 22:53:58.466719 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43752: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43605/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:53:58.466852 19793 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43752: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43605/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 22:53:58.468685 19809 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:58.462993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:43709 (local address 127.0.0.1:43752)
0928 22:53:58.463147 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:58.463158 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:58.463223 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:58.463385 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:58.463386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:58.463388 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 22:53:58.463434 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:58.463436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.464376 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.464377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:58.464973 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:58.464975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.465482 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.465483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:58.465523 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:58.465529 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 22:53:58.466690 (+ 1161us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43605/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 22:53:58.468593 (+ 1903us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.14.190:43709: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43605/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":1687,"thread_start_us":104,"threads_started":1}
I20260928 22:53:58.469074 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 19713
E20260928 22:53:58.479506 19810 webserver.cc:533] Webserver: sslize failed:
E20260928 22:53:58.479592 19810 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T22:53:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T22:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:58Z Disabled control of system clock
I20260928 22:53:58.514612 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:38813
--webserver_interface=127.14.14.190
--webserver_port=0
--builtin_ntp_servers=127.14.14.148:35805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:38813
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/testchainca.cert with env {}
W20260928 22:53:58.595503 19814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:58.595702 19814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:58.595726 19814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:58.596796 19814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:58.596832 19814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:58.596846 19814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:58.596859 19814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:58.596879 19814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 22:53:58.596892 19814 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 22:53:58.596904 19814 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 22:53:58.598834 19814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:35805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:38813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:38813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:58.599035 19814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:58.599193 19814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:58.604189 19820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:58.604193 19822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:58.604394 19819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:58.604533 19814 server_base.cc:1034] running on GCE node
I20260928 22:53:58.604712 19814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:58.604959 19814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:58.606110 19814 hybrid_clock.cc:648] HybridClock initialized: now 1790636038606092 us; error 29 us; skew 500 ppm
I20260928 22:53:58.607366 19814 webserver.cc:460] Webserver started at http://127.14.14.190:39519/ using document root <none> and password file <none>
I20260928 22:53:58.607604 19814 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:58.607650 19814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:58.607795 19814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:58.608841 19814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "9fdae42cad1b413783b0289db5a98ca7"
format_stamp: "Formatted at 2026-09-28 22:53:58 on dist-test-slave-kzfr"
server_key: "5dce32f783dde6643ce43ebbacb2ba8b"
server_key_iv: "f6cf000b46d2ab2b3d26d4b1f2df1c7a"
server_key_version: "encryptionkey@0"
I20260928 22:53:58.609210 19814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "9fdae42cad1b413783b0289db5a98ca7"
format_stamp: "Formatted at 2026-09-28 22:53:58 on dist-test-slave-kzfr"
server_key: "5dce32f783dde6643ce43ebbacb2ba8b"
server_key_iv: "f6cf000b46d2ab2b3d26d4b1f2df1c7a"
server_key_version: "encryptionkey@0"
I20260928 22:53:58.610610 19814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:58.611353 19828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:58.611541 19814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:58.611629 19814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "9fdae42cad1b413783b0289db5a98ca7"
format_stamp: "Formatted at 2026-09-28 22:53:58 on dist-test-slave-kzfr"
server_key: "5dce32f783dde6643ce43ebbacb2ba8b"
server_key_iv: "f6cf000b46d2ab2b3d26d4b1f2df1c7a"
server_key_version: "encryptionkey@0"
I20260928 22:53:58.611706 19814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:58.626027 19814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:58.626344 19814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:58.626495 19814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:58.630530 19814 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:38813
I20260928 22:53:58.630560 19880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:38813 every 8 connection(s)
I20260928 22:53:58.630934 19814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:58.631896 19881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:58.634294 19881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7: Bootstrap starting.
I20260928 22:53:58.634871 19881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:58.635154 19881 log.cc:824] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:58.635807 19881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7: No bootstrap required, opened a new log
I20260928 22:53:58.637383 19881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdae42cad1b413783b0289db5a98ca7" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38813 } }
I20260928 22:53:58.637521 19881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:58.637544 19881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fdae42cad1b413783b0289db5a98ca7, State: Initialized, Role: FOLLOWER
I20260928 22:53:58.637774 19881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdae42cad1b413783b0289db5a98ca7" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38813 } }
I20260928 22:53:58.637841 19881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:58.637863 19881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:58.637931 19881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:58.638476 19881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdae42cad1b413783b0289db5a98ca7" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38813 } }
I20260928 22:53:58.638587 19881 leader_election.cc:302] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fdae42cad1b413783b0289db5a98ca7; no voters:
I20260928 22:53:58.638782 19881 leader_election.cc:288] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:58.638816 19884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:58.638878 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 19814
I20260928 22:53:58.638939 19884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [term 1 LEADER]: Becoming Leader. State: Replica: 9fdae42cad1b413783b0289db5a98ca7, State: Running, Role: LEADER
I20260928 22:53:58.638988 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:53:58.639045 19881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:58.639037 19884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdae42cad1b413783b0289db5a98ca7" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38813 } }
I20260928 22:53:58.639187 14394 external_mini_cluster.cc:1428] Setting key 77e418dda9f7cc4e16ce1491869890a1
I20260928 22:53:58.639470 19886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fdae42cad1b413783b0289db5a98ca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdae42cad1b413783b0289db5a98ca7" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38813 } } }
I20260928 22:53:58.639568 19886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:58.639719 19886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fdae42cad1b413783b0289db5a98ca7. Latest consensus state: current_term: 1 leader_uuid: "9fdae42cad1b413783b0289db5a98ca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdae42cad1b413783b0289db5a98ca7" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38813 } } }
I20260928 22:53:58.639783 19886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:58.640410 19892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:58.641120 19892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:58.642630 19892 catalog_manager.cc:1347] Generated new cluster ID: 82ec451cbe5a4de2be76d3c230487b7b
I20260928 22:53:58.642683 19892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:58.644954 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:58.639703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.14.14.190:38813)
0928 22:53:58.639907 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:58.639916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:58.639952 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:53:58.640072 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:58.640079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:58.640150 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:58.640342 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:58.640588 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.641763 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.642835 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.643429 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.643536 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:58.644252 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:58.644256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:58.644257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:58.644468 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:58.644506 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:58.644519 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:58.644577 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:58.644676 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 22:53:58.644748 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260928 22:53:58.645992 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:58.663321 19892 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:58.663841 19892 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:58.669490 19892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fdae42cad1b413783b0289db5a98ca7: Generated new TSK 0
I20260928 22:53:58.669708 19892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:59.158937 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:58.677031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.14.14.190:38813)
0928 22:53:58.677077 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:58.677079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:58.677157 (+ 78us) server_negotiation.cc:407] Connection header received
0928 22:53:58.677234 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:58.677235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:58.677244 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:58.677302 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0928 22:53:58.677431 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.677971 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.679108 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.679283 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.679337 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:58.679514 (+ 177us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 22:53:59.158693 (+479179us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 22:53:59.158855 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43564: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 22:53:59.159003 19895 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43564: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 22:53:59.159132 19910 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:58.676933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38813 (local address 127.0.0.1:43564)
0928 22:53:58.677085 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:58.677096 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:58.677165 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:58.677303 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:58.677304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:58.677306 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 22:53:58.677373 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:58.677374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.678046 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.678047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:58.679021 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:58.679024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:58.679361 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:58.679362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:58.679422 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:58.679427 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 22:53:59.158839 (+479412us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 22:53:59.158980 (+ 141us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.14.190:38813: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260928 22:53:59.159679 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 19814
2026-09-28T22:53:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (669 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:59 dist-test-slave-kzfr krb5kdc[19922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:59 dist-test-slave-kzfr krb5kdc[19922](info): set up 2 sockets
Sep 28 22:53:59 dist-test-slave-kzfr krb5kdc[19922](info): commencing operation
krb5kdc: starting...
W20260928 22:54:00.370136 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:54:00 dist-test-slave-kzfr krb5kdc[19922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:54:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:54:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:54:00.489053 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:46229
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:46229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:00.568441 19935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:00.568650 19935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:00.568673 19935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:00.569891 19935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:54:00.569936 19935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:00.569949 19935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:00.569963 19935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:54:00.570024 19935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:54:00.571501 19935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:46229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:46229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:00.571765 19935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:00.571997 19935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:00.577000 19941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:00.577006 19943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:00.577021 19940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:00.577080 19935 server_base.cc:1034] running on GCE node
I20260928 22:54:00.577414 19935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:00.577728 19935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:00.578868 19935 hybrid_clock.cc:648] HybridClock initialized: now 1790636040578848 us; error 37 us; skew 500 ppm
Sep 28 22:54:00 dist-test-slave-kzfr krb5kdc[19922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636040, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:00.581372 19935 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:54:00.581859 19935 webserver.cc:460] Webserver started at http://127.14.14.190:40765/ using document root <none> and password file <none>
I20260928 22:54:00.582103 19935 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:00.582149 19935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:00.582295 19935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:00.583338 19935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "38b0054db34c46008840b818988bd22c"
format_stamp: "Formatted at 2026-09-28 22:54:00 on dist-test-slave-kzfr"
server_key: "ad4c039c8b35c60e3e27a4b9818e7bd3"
server_key_iv: "218010161f47cc0561f07d86edd75a60"
server_key_version: "encryptionkey@0"
I20260928 22:54:00.583698 19935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "38b0054db34c46008840b818988bd22c"
format_stamp: "Formatted at 2026-09-28 22:54:00 on dist-test-slave-kzfr"
server_key: "ad4c039c8b35c60e3e27a4b9818e7bd3"
server_key_iv: "218010161f47cc0561f07d86edd75a60"
server_key_version: "encryptionkey@0"
I20260928 22:54:00.585001 19935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:54:00.585729 19950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:00.585898 19935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:00.586009 19935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "38b0054db34c46008840b818988bd22c"
format_stamp: "Formatted at 2026-09-28 22:54:00 on dist-test-slave-kzfr"
server_key: "ad4c039c8b35c60e3e27a4b9818e7bd3"
server_key_iv: "218010161f47cc0561f07d86edd75a60"
server_key_version: "encryptionkey@0"
I20260928 22:54:00.586091 19935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:00.607470 19935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:00.609795 19935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:00.609972 19935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:00.613832 19935 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:46229
I20260928 22:54:00.613883 20002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:46229 every 8 connection(s)
I20260928 22:54:00.614262 19935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:54:00.615079 20003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:00.617275 20003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c: Bootstrap starting.
I20260928 22:54:00.617918 20003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:00.618207 20003 log.cc:824] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:00.618863 20003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c: No bootstrap required, opened a new log
I20260928 22:54:00.620424 20003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b0054db34c46008840b818988bd22c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46229 } }
I20260928 22:54:00.620558 20003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:00.620579 20003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b0054db34c46008840b818988bd22c, State: Initialized, Role: FOLLOWER
I20260928 22:54:00.620772 20003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b0054db34c46008840b818988bd22c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46229 } }
I20260928 22:54:00.620836 20003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:54:00.620858 20003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:54:00.620931 20003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:00.621469 20003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b0054db34c46008840b818988bd22c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46229 } }
I20260928 22:54:00.621580 20003 leader_election.cc:302] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38b0054db34c46008840b818988bd22c; no voters:
I20260928 22:54:00.621799 20003 leader_election.cc:288] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:54:00.621870 20006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:54:00.622062 20003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:54:00.622308 20006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [term 1 LEADER]: Becoming Leader. State: Replica: 38b0054db34c46008840b818988bd22c, State: Running, Role: LEADER
I20260928 22:54:00.622463 20006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b0054db34c46008840b818988bd22c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46229 } }
I20260928 22:54:00.622915 20008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38b0054db34c46008840b818988bd22c. Latest consensus state: current_term: 1 leader_uuid: "38b0054db34c46008840b818988bd22c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b0054db34c46008840b818988bd22c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46229 } } }
I20260928 22:54:00.623010 20008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:00.622923 20007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38b0054db34c46008840b818988bd22c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b0054db34c46008840b818988bd22c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46229 } } }
I20260928 22:54:00.623250 20015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:54:00.623296 20007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:00.623764 20015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:54:00.623936 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 19935
I20260928 22:54:00.624046 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:54:00.624199 14394 external_mini_cluster.cc:1428] Setting key 876629b6a11fec24140d8e93aba451f9
I20260928 22:54:00.625447 20015 catalog_manager.cc:1347] Generated new cluster ID: f8940340b6ba413f8dfe03efef2aaeee
I20260928 22:54:00.625500 20015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:54:00 dist-test-slave-kzfr krb5kdc[19922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:54:00.634371 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:00.624850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52062 (local address 127.14.14.190:46229)
0928 22:54:00.625036 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:00.625045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:00.625083 (+ 38us) server_negotiation.cc:407] Connection header received
0928 22:54:00.625198 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:00.625205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:00.625261 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:00.625708 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:00.626365 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:00.627274 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:00.628047 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:00.628473 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:00.628601 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:00.631001 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:00.631005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:00.631005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:00.631019 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:00.632716 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:00.633109 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:00.633112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:00.633112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:00.633204 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:00.633433 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:00.633434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:00.633434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:00.633739 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:00.633907 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:00.634002 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 22:54:00.634145 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260928 22:54:00.652002 20015 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:54:00.652591 20015 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:54:00.654940 20015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38b0054db34c46008840b818988bd22c: Generated new TSK 0
I20260928 22:54:00.655118 20015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:54:00.688311 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46229
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:00.784327 20029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:00.784538 20029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:00.784560 20029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:00.785697 20029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:00.785738 20029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:00.785765 20029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:54:00.787163 20029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:00.787343 20029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:00.787518 20029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:00.792714 20034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:00.792695 20035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:00.792752 20037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:00.793053 20029 server_base.cc:1034] running on GCE node
I20260928 22:54:00.793311 20029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:00.793517 20029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:00.794655 20029 hybrid_clock.cc:648] HybridClock initialized: now 1790636040794643 us; error 28 us; skew 500 ppm
Sep 28 22:54:00 dist-test-slave-kzfr krb5kdc[19922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636040, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:00.797273 20029 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:54:00.797772 20029 webserver.cc:460] Webserver started at http://127.14.14.129:39153/ using document root <none> and password file <none>
I20260928 22:54:00.797969 20029 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:00.798010 20029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:00.798098 20029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:00.799074 20029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "ed954aad22c3447ebf1239bc66b41973"
format_stamp: "Formatted at 2026-09-28 22:54:00 on dist-test-slave-kzfr"
server_key: "d95bb651c27d99b67b07ad827fb71405"
server_key_iv: "ed827dce823a60ecbe7edd1e40b4cab9"
server_key_version: "encryptionkey@0"
I20260928 22:54:00.799366 20029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "ed954aad22c3447ebf1239bc66b41973"
format_stamp: "Formatted at 2026-09-28 22:54:00 on dist-test-slave-kzfr"
server_key: "d95bb651c27d99b67b07ad827fb71405"
server_key_iv: "ed827dce823a60ecbe7edd1e40b4cab9"
server_key_version: "encryptionkey@0"
I20260928 22:54:00.800583 20029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:54:00.801263 20044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:00.801458 20029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:00.801555 20029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "ed954aad22c3447ebf1239bc66b41973"
format_stamp: "Formatted at 2026-09-28 22:54:00 on dist-test-slave-kzfr"
server_key: "d95bb651c27d99b67b07ad827fb71405"
server_key_iv: "ed827dce823a60ecbe7edd1e40b4cab9"
server_key_version: "encryptionkey@0"
I20260928 22:54:00.801651 20029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:00.828197 20029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:00.830221 20029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:00.830385 20029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:00.830626 20029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:00.830967 20029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:00.831032 20029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:00.831074 20029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:00.831102 20029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:00.836822 20029 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:33679
I20260928 22:54:00.836874 20157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:33679 every 8 connection(s)
I20260928 22:54:00.837230 20029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:54:00 dist-test-slave-kzfr krb5kdc[19922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636040, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:54:00.843957 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 20029
I20260928 22:54:00.844070 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:54:00.844276 14394 external_mini_cluster.cc:1428] Setting key f3719c7be857b39c512d87a8559d3e2f
I20260928 22:54:00.846400 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:00.838019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:42787 (local address 127.14.14.190:46229)
0928 22:54:00.838093 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:00.838095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:00.838286 (+ 191us) server_negotiation.cc:407] Connection header received
0928 22:54:00.839179 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:00.839180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:00.839190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:00.839239 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:00.839820 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:00.840325 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:00.841261 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:00.841454 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:00.841502 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:00.843432 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:00.843434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:00.843434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:00.843440 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:00.844900 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:00.845373 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:00.845374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:00.845375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:00.845440 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:00.845758 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:00.845759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:00.845759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:00.845873 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:00.845956 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:00.846226 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 22:54:00.846309 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 22:54:00.846611 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:00.837997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:46229 (local address 127.14.14.129:42787)
0928 22:54:00.838168 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:00.838203 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:00.839089 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:00.839301 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:00.839304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:00.839554 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:00.839747 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:00.839752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:00.840402 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:00.840403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:00.841186 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:00.841188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:00.841521 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:00.841522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:00.841640 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:00.842056 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:00.842064 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:00.843340 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:00.845014 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:00.845021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:00.845023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:00.845249 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:00.845509 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:00.845510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:00.845510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:00.845676 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:00.845946 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:00.845953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:00.846093 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 22:54:00.846204 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 22:54:00.846360 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260928 22:54:00.847100 20158 heartbeater.cc:347] Connected to a master server at 127.14.14.190:46229
I20260928 22:54:00.847189 20158 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:00.847371 20158 heartbeater.cc:511] Master 127.14.14.190:46229 requested a full tablet report, sending...
I20260928 22:54:00.847923 19967 ts_manager.cc:194] Registered new tserver with Master: ed954aad22c3447ebf1239bc66b41973 (127.14.14.129:33679)
I20260928 22:54:00.848856 19967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:42787
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:54:00.868276 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46229
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:00.947715 20163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:00.947938 20163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:00.947975 20163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:00.949126 20163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:00.949190 20163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:00.949235 20163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:54:00.950695 20163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:00.950965 20163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:00.951184 20163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:00.956207 20169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:00.956216 20168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:00.956216 20171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:00.956360 20163 server_base.cc:1034] running on GCE node
I20260928 22:54:00.956728 20163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:00.956928 20163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:00.958056 20163 hybrid_clock.cc:648] HybridClock initialized: now 1790636040958046 us; error 28 us; skew 500 ppm
Sep 28 22:54:00 dist-test-slave-kzfr krb5kdc[19922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636040, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:00.960371 20163 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:54:00.960829 20163 webserver.cc:460] Webserver started at http://127.14.14.130:33379/ using document root <none> and password file <none>
I20260928 22:54:00.961050 20163 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:00.961091 20163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:00.961236 20163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:00.962306 20163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "29908b8a68904f72895159cb4e9c6157"
format_stamp: "Formatted at 2026-09-28 22:54:00 on dist-test-slave-kzfr"
server_key: "e79b49a6e6245c119153a13475550f66"
server_key_iv: "228ad844ced6547d3423525ed9bdb891"
server_key_version: "encryptionkey@0"
I20260928 22:54:00.962658 20163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "29908b8a68904f72895159cb4e9c6157"
format_stamp: "Formatted at 2026-09-28 22:54:00 on dist-test-slave-kzfr"
server_key: "e79b49a6e6245c119153a13475550f66"
server_key_iv: "228ad844ced6547d3423525ed9bdb891"
server_key_version: "encryptionkey@0"
I20260928 22:54:00.963899 20163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:54:00.964577 20178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:00.964749 20163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:54:00.964844 20163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "29908b8a68904f72895159cb4e9c6157"
format_stamp: "Formatted at 2026-09-28 22:54:00 on dist-test-slave-kzfr"
server_key: "e79b49a6e6245c119153a13475550f66"
server_key_iv: "228ad844ced6547d3423525ed9bdb891"
server_key_version: "encryptionkey@0"
I20260928 22:54:00.964929 20163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:00.982976 20163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:00.984925 20163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:00.985093 20163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:00.985342 20163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:00.985739 20163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:00.985793 20163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:00.985833 20163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:00.985862 20163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:00.992177 20163 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:35015
I20260928 22:54:00.992237 20291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:35015 every 8 connection(s)
I20260928 22:54:00.992553 20163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:54:00.993073 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 20163
I20260928 22:54:00.993161 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:54:00.993407 14394 external_mini_cluster.cc:1428] Setting key cdb1638ccc0e763bbb798b1e5f7f254c
Sep 28 22:54:00 dist-test-slave-kzfr krb5kdc[19922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636040, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
I20260928 22:54:01.003260 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:00.993282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:60223 (local address 127.14.14.190:46229)
0928 22:54:00.993339 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:00.993342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:00.993629 (+ 287us) server_negotiation.cc:407] Connection header received
0928 22:54:00.994958 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:00.994959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:00.994970 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:00.995029 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:00.995913 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:00.996413 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:00.997424 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:00.997739 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:00.997829 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:01.000280 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:01.000282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:01.000282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:01.000290 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:01.001775 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:01.002241 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:01.002243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:01.002243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:01.002308 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:01.002579 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:01.002580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:01.002580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:01.002706 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:01.002779 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:01.003067 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 22:54:01.003155 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:54:01.003505 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:00.993288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:46229 (local address 127.14.14.130:60223)
0928 22:54:00.993463 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:00.993515 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:00.994832 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:00.995103 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:00.995107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:00.995515 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:00.995813 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:00.995818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:00.996509 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:00.996510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:00.997331 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:00.997333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:00.997837 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:00.997838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:00.997977 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:00.998536 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:00.998544 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:01.000178 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:01.001854 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:01.001862 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:01.001864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:01.002159 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:01.002375 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:01.002376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:01.002376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:01.002495 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:01.002779 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:01.002786 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:01.002935 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 22:54:01.003048 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 22:54:01.003253 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260928 22:54:01.004038 20292 heartbeater.cc:347] Connected to a master server at 127.14.14.190:46229
I20260928 22:54:01.004148 20292 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:01.004350 20292 heartbeater.cc:511] Master 127.14.14.190:46229 requested a full tablet report, sending...
I20260928 22:54:01.004832 19967 ts_manager.cc:194] Registered new tserver with Master: 29908b8a68904f72895159cb4e9c6157 (127.14.14.130:35015)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
I20260928 22:54:01.005275 19967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:60223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:54:01.021529 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46229
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:01.108441 20297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:01.108704 20297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:01.108752 20297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:01.110093 20297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:01.110167 20297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:01.110216 20297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:54:01.111866 20297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:01.112179 20297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:01.112423 20297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:01.117772 20303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:01.117861 20302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:01.117772 20305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:01.118031 20297 server_base.cc:1034] running on GCE node
I20260928 22:54:01.118265 20297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:01.118502 20297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:01.119647 20297 hybrid_clock.cc:648] HybridClock initialized: now 1790636041119631 us; error 28 us; skew 500 ppm
Sep 28 22:54:01 dist-test-slave-kzfr krb5kdc[19922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:01.122315 20297 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:54:01.122838 20297 webserver.cc:460] Webserver started at http://127.14.14.131:43131/ using document root <none> and password file <none>
I20260928 22:54:01.123100 20297 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:01.123170 20297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:01.123318 20297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:01.124414 20297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "a4d322f6849744da8a3ba57171c29dd9"
format_stamp: "Formatted at 2026-09-28 22:54:01 on dist-test-slave-kzfr"
server_key: "1be6f1d0c073fc2a241ee3a9df8f02f4"
server_key_iv: "04d6dea31f6299b3ed154a27787b7d0b"
server_key_version: "encryptionkey@0"
I20260928 22:54:01.124776 20297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "a4d322f6849744da8a3ba57171c29dd9"
format_stamp: "Formatted at 2026-09-28 22:54:01 on dist-test-slave-kzfr"
server_key: "1be6f1d0c073fc2a241ee3a9df8f02f4"
server_key_iv: "04d6dea31f6299b3ed154a27787b7d0b"
server_key_version: "encryptionkey@0"
I20260928 22:54:01.126200 20297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:54:01.127130 20312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:01.127307 20297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:01.127378 20297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "a4d322f6849744da8a3ba57171c29dd9"
format_stamp: "Formatted at 2026-09-28 22:54:01 on dist-test-slave-kzfr"
server_key: "1be6f1d0c073fc2a241ee3a9df8f02f4"
server_key_iv: "04d6dea31f6299b3ed154a27787b7d0b"
server_key_version: "encryptionkey@0"
I20260928 22:54:01.127480 20297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:01.138479 20297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:01.140293 20297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:01.140455 20297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:01.140676 20297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:01.140993 20297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:01.141026 20297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:01.141068 20297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:01.141096 20297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:01.146667 20297 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:35469
I20260928 22:54:01.146744 20425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:35469 every 8 connection(s)
I20260928 22:54:01.147050 20297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:54:01 dist-test-slave-kzfr krb5kdc[19922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:54:01.156055 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:01.147784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:60417 (local address 127.14.14.190:46229)
0928 22:54:01.147863 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:01.147865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:01.148023 (+ 158us) server_negotiation.cc:407] Connection header received
0928 22:54:01.148885 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:01.148886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:01.148897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:01.148946 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:01.149493 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:01.149992 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:01.150921 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:01.151095 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:01.151144 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:01.153285 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:01.153287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:01.153287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:01.153294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:01.154683 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:01.155113 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:01.155114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:01.155115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:01.155174 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:01.155426 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:01.155427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:01.155427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:01.155545 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:01.155622 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:01.155878 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 22:54:01.155961 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 22:54:01.156256 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:01.147742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:46229 (local address 127.14.14.131:60417)
0928 22:54:01.147899 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:01.147935 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:01.148802 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:01.149008 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:01.149011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:01.149242 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:01.149427 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:01.149431 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:01.150065 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:01.150066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:01.150846 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:01.150848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:01.151163 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:01.151164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:01.151265 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:01.151770 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:01.151778 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:01.153195 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:01.154759 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:01.154765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:01.154767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:01.155016 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:01.155242 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:01.155242 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:01.155243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:01.155352 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:01.155613 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:01.155619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:01.155748 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 22:54:01.155855 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 22:54:01.156021 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260928 22:54:01.156550 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 20297
I20260928 22:54:01.156625 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:54:01.156757 20426 heartbeater.cc:347] Connected to a master server at 127.14.14.190:46229
I20260928 22:54:01.156785 14394 external_mini_cluster.cc:1428] Setting key 31ccdbfaea59d6000e34c983f5a528de
I20260928 22:54:01.156878 20426 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:01.157078 20426 heartbeater.cc:511] Master 127.14.14.190:46229 requested a full tablet report, sending...
I20260928 22:54:01.157469 19966 ts_manager.cc:194] Registered new tserver with Master: a4d322f6849744da8a3ba57171c29dd9 (127.14.14.131:35469)
I20260928 22:54:01.157903 19966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:60417
I20260928 22:54:01.168514 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/test-admin.keytab.
Sep 28 22:54:01 dist-test-slave-kzfr krb5kdc[19922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:01.185904 14394 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 22:54:01.187714 20433 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:01.186930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.14.14.129:33679)
0928 22:54:01.187116 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:01.187121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:01.187129 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:54:01.187154 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:01.187156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:01.187168 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:01.187251 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:01.187613 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41000: BlockingRecv error: recv got EOF from 127.0.0.1:41000 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
W20260928 22:54:01.187947 14394 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/krb5cc' of type FILE
Sep 28 22:54:01 dist-test-slave-kzfr krb5kdc[19922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:01.188938 14394 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 22:54:01 dist-test-slave-kzfr krb5kdc[19922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:01.189901 14394 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 22:54:01 dist-test-slave-kzfr krb5kdc[19922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.129@KRBTEST.COM
I20260928 22:54:01.197527 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:01.190393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41012 (local address 127.14.14.129:33679)
0928 22:54:01.190433 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:01.190435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:01.190470 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:54:01.190525 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:01.190526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:01.190534 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:01.190579 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:01.190938 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:01.191636 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:01.192322 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:01.192738 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:01.192793 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:01.194867 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:01.194874 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:01.194874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:01.194880 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:01.196612 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:01.196912 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:01.196915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:01.196915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:01.196968 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:01.197116 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:01.197117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:01.197117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:01.197237 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:01.197332 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:01.197397 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 22:54:01.197456 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 22:54:01.199600 20433 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:01.198903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.14.14.129:33679)
0928 22:54:01.198967 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:01.198969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:01.199094 (+ 125us) server_negotiation.cc:407] Connection header received
0928 22:54:01.199141 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:01.199142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:01.199152 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:01.199210 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:01.199553 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41028: BlockingRecv error: recv got EOF from 127.0.0.1:41028 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 22:54:01.199676 14394 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/krb5cc' of type FILE
Sep 28 22:54:01 dist-test-slave-kzfr krb5kdc[19922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:01.200721 14394 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 22:54:01 dist-test-slave-kzfr krb5kdc[19922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:01.201866 14394 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 22:54:01 dist-test-slave-kzfr krb5kdc[19922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.129@KRBTEST.COM
I20260928 22:54:01.209004 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:01.202390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41036 (local address 127.14.14.129:33679)
0928 22:54:01.202444 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:01.202446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:01.202457 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:54:01.202520 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:01.202520 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:01.202527 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:01.202571 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:01.202965 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:01.203469 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:01.204154 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:01.204346 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:01.204397 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:01.206498 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:01.206500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:01.206500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:01.206507 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:01.207857 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:01.208238 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:01.208240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:01.208240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:01.208296 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:01.208502 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:01.208503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:01.208504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:01.208601 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:01.208701 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:01.208816 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 22:54:01.208893 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 22:54:01.210839 20433 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:01.210175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41048 (local address 127.14.14.129:33679)
0928 22:54:01.210238 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:01.210239 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:01.210302 (+ 63us) server_negotiation.cc:407] Connection header received
0928 22:54:01.210376 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:01.210377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:01.210384 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:01.210434 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:01.210787 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41048: BlockingRecv error: recv got EOF from 127.0.0.1:41048 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 22:54:01.210984 14394 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790636016256058-14394-0/krb5kdc/krb5cc' of type FILE
Sep 28 22:54:01 dist-test-slave-kzfr krb5kdc[19922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:01.212023 14394 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 22:54:01 dist-test-slave-kzfr krb5kdc[19922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:01.212955 14394 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 22:54:01 dist-test-slave-kzfr krb5kdc[19922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.129@KRBTEST.COM
I20260928 22:54:01.220039 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:01.213475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.14.14.129:33679)
0928 22:54:01.213513 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:01.213519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:01.213682 (+ 163us) server_negotiation.cc:407] Connection header received
0928 22:54:01.213742 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:01.213743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:01.213750 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:01.213794 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:01.214231 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:01.214674 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:01.215358 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:01.215540 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:01.215594 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:01.217481 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:01.217485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:01.217485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:01.217497 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:01.218903 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:01.219264 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:01.219266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:01.219266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:01.219327 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:01.219542 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:01.219543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:01.219543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:01.219642 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:01.219741 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:01.219854 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 22:54:01.219934 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:54:01.220775 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 20029
I20260928 22:54:01.225570 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 20163
I20260928 22:54:01.229979 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 20297
I20260928 22:54:01.233876 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 19935
2026-09-28T22:54:01Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1900 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:54:01 dist-test-slave-kzfr krb5kdc[20446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:54:01 dist-test-slave-kzfr krb5kdc[20446](info): set up 2 sockets
Sep 28 22:54:01 dist-test-slave-kzfr krb5kdc[20446](info): commencing operation
krb5kdc: starting...
W20260928 22:54:02.282078 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:54:02 dist-test-slave-kzfr krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:54:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:54:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.190.keytab.
Entry for principal oryx/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.190.keytab.
I20260928 22:54:02.405398 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:35807
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.190.keytab
--principal=oryx/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:35807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:02.487538 20459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:02.487795 20459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:02.487833 20459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:02.489675 20459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:54:02.489745 20459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:02.489769 20459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:02.489809 20459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:54:02.489836 20459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:54:02.491444 20459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:35807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.190.keytab
--principal=oryx/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:35807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:02.491742 20459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:02.491977 20459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:02.496899 20464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:02.496923 20465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:02.497093 20467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:02.497131 20459 server_base.cc:1034] running on GCE node
I20260928 22:54:02.497411 20459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:02.497768 20459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:02.498930 20459 hybrid_clock.cc:648] HybridClock initialized: now 1790636042498917 us; error 42 us; skew 500 ppm
Sep 28 22:54:02 dist-test-slave-kzfr krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636042, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:02.501457 20459 init.cc:377] Logged in from keytab as oryx/127.14.14.190@KRBTEST.COM (short username oryx)
I20260928 22:54:02.501979 20459 webserver.cc:460] Webserver started at http://127.14.14.190:44687/ using document root <none> and password file <none>
I20260928 22:54:02.502220 20459 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:02.502288 20459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:02.502450 20459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:02.503640 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "bca8e77469ae4f95a452cf2b744ddfdf"
format_stamp: "Formatted at 2026-09-28 22:54:02 on dist-test-slave-kzfr"
server_key: "6d42a2af59ef2984dca95ab65d4f8a33"
server_key_iv: "bab02db70f048a153ce4f1fafb0dc684"
server_key_version: "encryptionkey@0"
I20260928 22:54:02.504091 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "bca8e77469ae4f95a452cf2b744ddfdf"
format_stamp: "Formatted at 2026-09-28 22:54:02 on dist-test-slave-kzfr"
server_key: "6d42a2af59ef2984dca95ab65d4f8a33"
server_key_iv: "bab02db70f048a153ce4f1fafb0dc684"
server_key_version: "encryptionkey@0"
I20260928 22:54:02.505412 20459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:54:02.506208 20474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:02.506385 20459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:54:02.506500 20459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "bca8e77469ae4f95a452cf2b744ddfdf"
format_stamp: "Formatted at 2026-09-28 22:54:02 on dist-test-slave-kzfr"
server_key: "6d42a2af59ef2984dca95ab65d4f8a33"
server_key_iv: "bab02db70f048a153ce4f1fafb0dc684"
server_key_version: "encryptionkey@0"
I20260928 22:54:02.506585 20459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:02.518038 20459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:02.520298 20459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:02.520465 20459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:02.524653 20459 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:35807
I20260928 22:54:02.524703 20526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:35807 every 8 connection(s)
I20260928 22:54:02.525076 20459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:54:02.526048 20527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:02.528220 20527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf: Bootstrap starting.
I20260928 22:54:02.528731 20527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:02.528964 20527 log.cc:824] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:02.529628 20527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf: No bootstrap required, opened a new log
I20260928 22:54:02.530359 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 20459
I20260928 22:54:02.530485 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:54:02.530689 14394 external_mini_cluster.cc:1428] Setting key 4768888573c503aef683709c7765a019
I20260928 22:54:02.531770 20527 raft_consensus.cc:348] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } }
I20260928 22:54:02.531937 20527 raft_consensus.cc:374] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:02.531987 20527 raft_consensus.cc:729] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca8e77469ae4f95a452cf2b744ddfdf, State: Initialized, Role: FOLLOWER
I20260928 22:54:02.532116 20527 consensus_queue.cc:260] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } }
I20260928 22:54:02.532202 20527 raft_consensus.cc:388] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:54:02.532238 20527 raft_consensus.cc:482] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:54:02.532286 20527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:02.533254 20527 raft_consensus.cc:504] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } }
I20260928 22:54:02.533344 20527 leader_election.cc:302] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bca8e77469ae4f95a452cf2b744ddfdf; no voters:
I20260928 22:54:02.533520 20527 leader_election.cc:288] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:54:02.533567 20532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:54:02.533722 20532 raft_consensus.cc:686] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [term 1 LEADER]: Becoming Leader. State: Replica: bca8e77469ae4f95a452cf2b744ddfdf, State: Running, Role: LEADER
I20260928 22:54:02.533802 20527 sys_catalog.cc:564] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:54:02.533802 20532 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } }
I20260928 22:54:02.534193 20533 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } }
I20260928 22:54:02.534287 20533 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:02.534539 20534 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca8e77469ae4f95a452cf2b744ddfdf. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } }
I20260928 22:54:02.534655 20534 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:02.534817 20539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:54:02.535387 20539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:54:02.537104 20539 catalog_manager.cc:1347] Generated new cluster ID: f7c612814c4549cf84628db94f6e15d2
I20260928 22:54:02.537160 20539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:54:02 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
I20260928 22:54:02.542399 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:02.531228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.14.14.190:35807)
0928 22:54:02.531438 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:02.531449 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:02.531476 (+ 27us) server_negotiation.cc:407] Connection header received
0928 22:54:02.531759 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:02.531765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:02.531811 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:02.531964 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:02.532738 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:02.533895 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:02.535017 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:02.535625 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:02.535797 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:02.538668 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:02.538672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:02.538672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:02.538684 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:02.540588 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:02.540968 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:02.540971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:02.540971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:02.541083 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:02.541259 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:02.541259 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:02.541259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:02.541530 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:02.541693 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:02.542073 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 22:54:02.542198 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
I20260928 22:54:02.566705 20539 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:54:02.567281 20539 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:54:02.575019 20539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf: Generated new TSK 0
I20260928 22:54:02.575228 20539 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.129.keytab.
Entry for principal oryx/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.129.keytab.
I20260928 22:54:02.612643 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:35807
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.129.keytab
--principal=oryx/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:02.706979 20553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:02.707171 20553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:02.707193 20553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:02.708256 20553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:02.708292 20553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:02.708317 20553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:54:02.709718 20553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.129.keytab
--principal=oryx/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:35807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:02.709895 20553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:02.710063 20553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:02.715047 20558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:02.715066 20559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:02.715066 20561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:02.715140 20553 server_base.cc:1034] running on GCE node
I20260928 22:54:02.715461 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:02.715688 20553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:02.716846 20553 hybrid_clock.cc:648] HybridClock initialized: now 1790636042716842 us; error 34 us; skew 500 ppm
Sep 28 22:54:02 dist-test-slave-kzfr krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636042, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:02.719373 20553 init.cc:377] Logged in from keytab as oryx/127.14.14.129@KRBTEST.COM (short username oryx)
I20260928 22:54:02.719789 20553 webserver.cc:460] Webserver started at http://127.14.14.129:35021/ using document root <none> and password file <none>
I20260928 22:54:02.720033 20553 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:02.720104 20553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:02.720248 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:02.721284 20553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "ae618bb3e59841e0a60be96d1b74235e"
format_stamp: "Formatted at 2026-09-28 22:54:02 on dist-test-slave-kzfr"
server_key: "a4395805f290b68a4b46ab24606a4058"
server_key_iv: "4d6f720e8b4979738c5621d686f24049"
server_key_version: "encryptionkey@0"
I20260928 22:54:02.721690 20553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "ae618bb3e59841e0a60be96d1b74235e"
format_stamp: "Formatted at 2026-09-28 22:54:02 on dist-test-slave-kzfr"
server_key: "a4395805f290b68a4b46ab24606a4058"
server_key_iv: "4d6f720e8b4979738c5621d686f24049"
server_key_version: "encryptionkey@0"
I20260928 22:54:02.722990 20553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:02.723685 20568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:02.723845 20553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:02.723966 20553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "ae618bb3e59841e0a60be96d1b74235e"
format_stamp: "Formatted at 2026-09-28 22:54:02 on dist-test-slave-kzfr"
server_key: "a4395805f290b68a4b46ab24606a4058"
server_key_iv: "4d6f720e8b4979738c5621d686f24049"
server_key_version: "encryptionkey@0"
I20260928 22:54:02.724042 20553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:02.745393 20553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:02.747751 20553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:02.747916 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:02.748154 20553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:02.748458 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:02.748512 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:02.748555 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:02.748584 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:02.754276 20553 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:45887
I20260928 22:54:02.754352 20681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:45887 every 8 connection(s)
I20260928 22:54:02.754673 20553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:54:02.758203 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 20553
I20260928 22:54:02.758303 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:54:02.758496 14394 external_mini_cluster.cc:1428] Setting key 8e13722fd8ba9ca0616c810e4a406a72
Sep 28 22:54:02 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636042, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
I20260928 22:54:02.764868 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:02.755391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:57935 (local address 127.14.14.190:35807)
0928 22:54:02.755465 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:02.755467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:02.755644 (+ 177us) server_negotiation.cc:407] Connection header received
0928 22:54:02.756106 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:02.756107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:02.756116 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:02.756168 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:02.757058 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:02.757553 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:02.758955 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:02.759211 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:02.759267 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:02.761931 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:02.761946 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:02.761946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:02.761954 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:02.763381 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:02.763896 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:02.763897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:02.763897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:02.763960 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:02.764252 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:02.764252 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:02.764252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:02.764347 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:02.764396 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:02.764738 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 22:54:02.764805 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:54:02.765153 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:02.755353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.129:57935)
0928 22:54:02.755516 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:02.755557 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:02.756028 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:02.756228 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:02.756231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:02.756787 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:02.756982 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:02.756987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:02.757688 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:02.757689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:02.758883 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:02.758885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:02.759279 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:02.759280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:02.759430 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:02.759909 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:02.759920 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:02.761838 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:02.763467 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:02.763474 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:02.763477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:02.763811 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:02.764028 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:02.764029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:02.764029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:02.764181 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:02.764415 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:02.764422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:02.764594 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 22:54:02.764724 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 22:54:02.764892 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260928 22:54:02.765722 20682 heartbeater.cc:347] Connected to a master server at 127.14.14.190:35807
I20260928 22:54:02.765827 20682 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:02.766031 20682 heartbeater.cc:511] Master 127.14.14.190:35807 requested a full tablet report, sending...
I20260928 22:54:02.766547 20491 ts_manager.cc:194] Registered new tserver with Master: ae618bb3e59841e0a60be96d1b74235e (127.14.14.129:45887)
WARNING: no policy specified for oryx/127.14.14.130@KRBTEST.COM; defaulting to no policy
I20260928 22:54:02.767159 20491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:57935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.130.keytab.
Entry for principal oryx/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.130.keytab.
I20260928 22:54:02.785346 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:35807
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.130.keytab
--principal=oryx/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:02.865104 20687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:02.865304 20687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:02.865326 20687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:02.866514 20687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:02.866556 20687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:02.866582 20687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:54:02.868067 20687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.130.keytab
--principal=oryx/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:35807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:02.868309 20687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:02.868539 20687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:02.873716 20692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:02.873803 20693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:02.873827 20687 server_base.cc:1034] running on GCE node
W20260928 22:54:02.873764 20695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:02.874120 20687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:02.874354 20687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:02.875512 20687 hybrid_clock.cc:648] HybridClock initialized: now 1790636042875503 us; error 45 us; skew 500 ppm
Sep 28 22:54:02 dist-test-slave-kzfr krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636042, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:02.877898 20687 init.cc:377] Logged in from keytab as oryx/127.14.14.130@KRBTEST.COM (short username oryx)
I20260928 22:54:02.878368 20687 webserver.cc:460] Webserver started at http://127.14.14.130:35967/ using document root <none> and password file <none>
I20260928 22:54:02.878592 20687 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:02.878661 20687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:02.878810 20687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:02.879802 20687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "c89c8d94bf79445189e99922cdefdef8"
format_stamp: "Formatted at 2026-09-28 22:54:02 on dist-test-slave-kzfr"
server_key: "f934d7809d688736af00b61894e7d679"
server_key_iv: "032e2c182583ec233c49c41e955b2bf4"
server_key_version: "encryptionkey@0"
I20260928 22:54:02.880160 20687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "c89c8d94bf79445189e99922cdefdef8"
format_stamp: "Formatted at 2026-09-28 22:54:02 on dist-test-slave-kzfr"
server_key: "f934d7809d688736af00b61894e7d679"
server_key_iv: "032e2c182583ec233c49c41e955b2bf4"
server_key_version: "encryptionkey@0"
I20260928 22:54:02.881479 20687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:54:02.882205 20702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:02.882375 20687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:02.882469 20687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "c89c8d94bf79445189e99922cdefdef8"
format_stamp: "Formatted at 2026-09-28 22:54:02 on dist-test-slave-kzfr"
server_key: "f934d7809d688736af00b61894e7d679"
server_key_iv: "032e2c182583ec233c49c41e955b2bf4"
server_key_version: "encryptionkey@0"
I20260928 22:54:02.882551 20687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:02.892386 20687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:02.894268 20687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:02.894424 20687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:02.894631 20687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:02.894927 20687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:02.894995 20687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:02.895036 20687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:02.895066 20687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:02.901001 20687 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:42869
I20260928 22:54:02.901053 20815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:42869 every 8 connection(s)
I20260928 22:54:02.901377 20687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:54:02 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636042, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
I20260928 22:54:02.910108 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 20687
I20260928 22:54:02.910219 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:54:02.910218 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:02.902151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:47041 (local address 127.14.14.190:35807)
0928 22:54:02.902222 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:02.902223 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:02.902408 (+ 185us) server_negotiation.cc:407] Connection header received
0928 22:54:02.902895 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:02.902897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:02.902906 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:02.902985 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:02.903876 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:02.904414 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:02.905337 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:02.905521 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:02.905570 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:02.907604 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:02.907605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:02.907605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:02.907610 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:02.908842 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:02.909208 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:02.909209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:02.909209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:02.909263 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:02.909469 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:02.909469 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:02.909470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:02.909571 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:02.909669 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:02.909975 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 22:54:02.910106 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:54:02.910427 14394 external_mini_cluster.cc:1428] Setting key d31efdaab742ad1c852a9c32becdfc53
I20260928 22:54:02.910465 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:02.902112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.130:47041)
0928 22:54:02.902280 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:02.902322 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:02.902814 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:02.903043 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:02.903046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:02.903603 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:02.903788 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:02.903792 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:02.904494 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:02.904495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:02.905267 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:02.905268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:02.905604 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:02.905605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:02.905700 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:02.906226 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:02.906234 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:02.907528 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:02.908914 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:02.908920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:02.908921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:02.909143 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:02.909314 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:02.909315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:02.909315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:02.909408 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:02.909678 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:02.909682 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:02.909809 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 22:54:02.909944 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 22:54:02.910129 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260928 22:54:02.910920 20816 heartbeater.cc:347] Connected to a master server at 127.14.14.190:35807
I20260928 22:54:02.911015 20816 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:02.911193 20816 heartbeater.cc:511] Master 127.14.14.190:35807 requested a full tablet report, sending...
I20260928 22:54:02.911643 20490 ts_manager.cc:194] Registered new tserver with Master: c89c8d94bf79445189e99922cdefdef8 (127.14.14.130:42869)
I20260928 22:54:02.912039 20490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:47041
WARNING: no policy specified for oryx/127.14.14.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.131.keytab.
Entry for principal oryx/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.131.keytab.
I20260928 22:54:02.936565 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:35807
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.131.keytab
--principal=oryx/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:03.015817 20821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:03.016036 20821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:03.016059 20821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:03.017133 20821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:03.017167 20821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:03.017192 20821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:54:03.018635 20821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.131.keytab
--principal=oryx/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:35807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:03.018824 20821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:03.019001 20821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:03.024034 20829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:03.024068 20827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:03.024147 20821 server_base.cc:1034] running on GCE node
W20260928 22:54:03.024030 20826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:03.024459 20821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:03.024712 20821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:03.025892 20821 hybrid_clock.cc:648] HybridClock initialized: now 1790636043025925 us; error 98 us; skew 500 ppm
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:03.028501 20821 init.cc:377] Logged in from keytab as oryx/127.14.14.131@KRBTEST.COM (short username oryx)
I20260928 22:54:03.028923 20821 webserver.cc:460] Webserver started at http://127.14.14.131:34151/ using document root <none> and password file <none>
I20260928 22:54:03.029152 20821 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:03.029223 20821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:03.029392 20821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:03.030623 20821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "5fb9e57173d94423a7e31e876116de53"
format_stamp: "Formatted at 2026-09-28 22:54:03 on dist-test-slave-kzfr"
server_key: "65a8670def992e2dd63b9b576f776bb5"
server_key_iv: "ce854afa34af983c9b426c3f7cd7c474"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.031019 20821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "5fb9e57173d94423a7e31e876116de53"
format_stamp: "Formatted at 2026-09-28 22:54:03 on dist-test-slave-kzfr"
server_key: "65a8670def992e2dd63b9b576f776bb5"
server_key_iv: "ce854afa34af983c9b426c3f7cd7c474"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.032334 20821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:54:03.033067 20836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.033280 20821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:03.033378 20821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "5fb9e57173d94423a7e31e876116de53"
format_stamp: "Formatted at 2026-09-28 22:54:03 on dist-test-slave-kzfr"
server_key: "65a8670def992e2dd63b9b576f776bb5"
server_key_iv: "ce854afa34af983c9b426c3f7cd7c474"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.033468 20821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:03.054631 20821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:03.056537 20821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:03.056710 20821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:03.056968 20821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:03.057279 20821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:03.057333 20821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.057374 20821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:03.057418 20821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.063377 20821 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:36527
I20260928 22:54:03.063421 20949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:36527 every 8 connection(s)
I20260928 22:54:03.063748 20821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
I20260928 22:54:03.071426 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 20821
I20260928 22:54:03.071539 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:54:03.071750 14394 external_mini_cluster.cc:1428] Setting key 4f824d27c5b30407fc11b17d455d419f
I20260928 22:54:03.072697 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.064498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:50397 (local address 127.14.14.190:35807)
0928 22:54:03.064541 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.064543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.064771 (+ 228us) server_negotiation.cc:407] Connection header received
0928 22:54:03.065237 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.065238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.065248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.065296 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.066240 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.066725 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.067661 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.067837 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.067887 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.069982 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.069983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.069983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.069990 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.071200 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.071688 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.071689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.071689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.071751 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.072053 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.072054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.072055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.072178 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.072259 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.072524 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.072607 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:54:03.072898 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.064497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.131:50397)
0928 22:54:03.064643 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.064678 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.065147 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.065359 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.065362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.065962 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.066164 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.066169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.066809 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.066810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.067585 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.067587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.067909 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.067909 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.068005 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.068463 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.068471 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.069882 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.071293 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.071300 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.071302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.071615 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.071830 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.071831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.071831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.071974 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.072249 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.072255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.072388 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.072503 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.072661 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260928 22:54:03.073319 20950 heartbeater.cc:347] Connected to a master server at 127.14.14.190:35807
I20260928 22:54:03.073397 20950 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:03.073642 20950 heartbeater.cc:511] Master 127.14.14.190:35807 requested a full tablet report, sending...
I20260928 22:54:03.074018 20490 ts_manager.cc:194] Registered new tserver with Master: 5fb9e57173d94423a7e31e876116de53 (127.14.14.131:36527)
I20260928 22:54:03.074404 20490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:50397
I20260928 22:54:03.083350 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.14.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.14.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.14.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.189.keytab.
Entry for principal oryx/127.14.14.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.189.keytab.
I20260928 22:54:03.109243 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.189:39427
--webserver_interface=127.14.14.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.189.keytab
--principal=oryx/127.14.14.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:35807,127.14.14.189:39427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:03.189527 20955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:03.189811 20955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:03.189850 20955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:03.190997 20955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:54:03.191066 20955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:03.191089 20955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:03.191120 20955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:54:03.191151 20955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:54:03.192595 20955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:35807,127.14.14.189:39427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.189.keytab
--principal=oryx/127.14.14.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.189:39427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.14.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
W20260928 22:54:03.192729 20955 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.14.190:35807,127.14.14.189:39427'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 22:54:03.192884 20955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:03.193109 20955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:54:03.198107 20955 server_base.cc:1034] running on GCE node
W20260928 22:54:03.198091 20963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:03.198243 20961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:03.198091 20960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:03.198535 20955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:03.198827 20955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:03.199973 20955 hybrid_clock.cc:648] HybridClock initialized: now 1790636043199957 us; error 41 us; skew 500 ppm
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:03.202468 20955 init.cc:377] Logged in from keytab as oryx/127.14.14.189@KRBTEST.COM (short username oryx)
I20260928 22:54:03.202836 20955 webserver.cc:460] Webserver started at http://127.14.14.189:46645/ using document root <none> and password file <none>
I20260928 22:54:03.203032 20955 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:03.203071 20955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:03.203157 20955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:03.204154 20955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/data/instance:
uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee"
format_stamp: "Formatted at 2026-09-28 22:54:03 on dist-test-slave-kzfr"
server_key: "457936a9f7b6b151034ccbefb7e0be63"
server_key_iv: "921c09d342d682f12e38fa3feaadf2df"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.204438 20955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/wal/instance:
uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee"
format_stamp: "Formatted at 2026-09-28 22:54:03 on dist-test-slave-kzfr"
server_key: "457936a9f7b6b151034ccbefb7e0be63"
server_key_iv: "921c09d342d682f12e38fa3feaadf2df"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.205801 20955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:54:03.206449 20970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.206606 20955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:03.206702 20955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/wal
uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee"
format_stamp: "Formatted at 2026-09-28 22:54:03 on dist-test-slave-kzfr"
server_key: "457936a9f7b6b151034ccbefb7e0be63"
server_key_iv: "921c09d342d682f12e38fa3feaadf2df"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.206780 20955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:03.229120 20955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:03.231535 20955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:03.231699 20955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:03.235857 20955 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.189:39427
I20260928 22:54:03.235911 21022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.189:39427 every 8 connection(s)
I20260928 22:54:03.236343 20955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/data/info.pb
I20260928 22:54:03.237380 21023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:03.238909 21023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:54:03.244007 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 20955
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.189@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
I20260928 22:54:03.247840 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.239584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43756 (local address 127.14.14.190:35807)
0928 22:54:03.239653 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.239655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.239864 (+ 209us) server_negotiation.cc:407] Connection header received
0928 22:54:03.240363 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.240364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.240376 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.240436 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.241059 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.241562 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.242542 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.242719 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.242771 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.244997 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.244999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.245000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.245006 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.246442 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.246813 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.246814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.246815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.246879 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.247099 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.247100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.247100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.247308 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.247359 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.247655 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.247746 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:54:03.248301 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.239556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:43756)
0928 22:54:03.239743 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.239783 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.240258 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.240502 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.240506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.240765 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.240956 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.240960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.241681 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.241682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.242471 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.242473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.242792 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.242792 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.242897 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.243378 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.243386 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.244916 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.246536 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.246541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.246542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.246743 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.246948 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.246948 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.246948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.247039 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.247367 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.247371 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.247525 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.247636 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.247819 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260928 22:54:03.248986 21023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.14.189@KRBTEST.COM
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.189@KRBTEST.COM for oryx/127.14.14.189@KRBTEST.COM
I20260928 22:54:03.253330 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.244521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.14.14.189:39427)
0928 22:54:03.244657 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.244661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.244719 (+ 58us) server_negotiation.cc:407] Connection header received
0928 22:54:03.244776 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.244779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.244799 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.244872 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.245396 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.246261 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.247264 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.247664 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.247734 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.250245 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.250253 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.250253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.250259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.252069 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.252433 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.252435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.252436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.252497 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.252757 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.252758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.252758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.252897 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.253065 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.253150 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.253225 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260928 22:54:03.255883 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.249228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:39427 (local address 127.0.0.1:52138)
0928 22:54:03.249276 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.249286 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.249331 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.249524 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.249525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.249697 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.249736 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.249737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.250452 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.250453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.250901 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.250903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.251629 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.251630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.251679 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.252064 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.252068 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.253345 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.255108 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.255108 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.255109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.255261 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.255471 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.255472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.255472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.255519 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.255687 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.255688 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.255718 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.255766 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.255831 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 22:54:03.256001 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.249264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.14.14.189:39427)
0928 22:54:03.249410 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.249411 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.249417 (+ 6us) server_negotiation.cc:407] Connection header received
0928 22:54:03.249433 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.249433 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.249441 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.249522 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.249811 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.250370 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.251222 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.251553 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.251626 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.253425 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.253425 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.253426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.253462 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.255039 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.255336 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.255337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.255337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.255397 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.255557 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.255558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.255558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.255640 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.255683 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.255779 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.255873 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260928 22:54:03.257073 21023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: Bootstrap starting.
I20260928 22:54:03.257735 21023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:03.258005 21023 log.cc:824] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:03.258670 21023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: No bootstrap required, opened a new log
I20260928 22:54:03.260102 21023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } }
I20260928 22:54:03.260237 21023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:03.260285 21023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f5bd5a76c94b4bbf6b5e099dfa46ee, State: Initialized, Role: FOLLOWER
I20260928 22:54:03.260425 21023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } }
I20260928 22:54:03.260658 21030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } } }
I20260928 22:54:03.260704 21023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:54:03.260773 21030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:54:03.262356 21041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:54:03.262398 21041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:54:03.263880 20955 master_runner.cc:428] Detected that this master 75f5bd5a76c94b4bbf6b5e099dfa46ee is joining an existing cluster
I20260928 22:54:03.263934 20955 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.14.189:39427
I20260928 22:54:03.271757 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.264963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.14.14.190:35807)
0928 22:54:03.265003 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.265005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.265335 (+ 330us) server_negotiation.cc:407] Connection header received
0928 22:54:03.265437 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.265438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.265456 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.265510 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.265880 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.266349 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.267458 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.267657 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.267749 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.268793 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.268794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.268794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.268801 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.270469 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.270905 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.270906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.270906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.270972 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.271205 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.271206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.271207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.271335 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.271377 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.271588 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.271660 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:54:03.271778 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.264931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:43764)
0928 22:54:03.265275 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.265281 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.265359 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.265525 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.265526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.265736 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.265789 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.265790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.266428 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.266429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.267345 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.267346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.267767 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.267768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.267829 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.268259 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.268264 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.268691 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.270558 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.270559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.270560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.270827 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.271046 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.271047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.271047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.271127 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.271434 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.271436 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.271475 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.271560 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.271671 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260928 22:54:03.271926 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.265166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.14.14.189:39427)
0928 22:54:03.265217 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.265219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.265226 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:54:03.265246 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.265247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.265260 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.265323 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.265750 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.266478 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.267427 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.267743 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.267814 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.268961 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.268962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.268962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.268968 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.270650 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.271018 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.271019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.271019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.271078 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.271282 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.271283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.271283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.271428 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.271501 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.271692 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.271779 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:54:03.272073 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.264970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:39427 (local address 127.0.0.1:52152)
0928 22:54:03.265139 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.265151 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.265200 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.265336 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.265337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.265570 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.265676 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.265678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.266557 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.266558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.267345 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.267346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.267860 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.267860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.267916 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.268438 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.268443 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.268869 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.270721 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.270722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.270723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.270945 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.271147 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.271147 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.271147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.271215 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.271525 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.271526 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.271560 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.271618 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.271910 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":196,"thread_start_us":110,"threads_started":1}
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636042, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.190@KRBTEST.COM for oryx/127.14.14.189@KRBTEST.COM
I20260928 22:54:03.281283 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.274680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.14.14.189:39427)
0928 22:54:03.274721 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.274723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.274842 (+ 119us) server_negotiation.cc:407] Connection header received
0928 22:54:03.274939 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.274940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.274949 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.274994 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.275404 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.275847 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.276621 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.276817 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.276897 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.278773 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.278775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.278775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.278782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.280079 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.280468 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.280469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.280469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.280526 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.280721 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.280722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.280722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.280842 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.280915 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.281114 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.281188 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:54:03.281298 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.274604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:39427 (local address 127.0.0.1:52160)
0928 22:54:03.274738 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.274783 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.274865 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.274997 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.275000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.275269 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.275315 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.275317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.275925 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.275926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.276519 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.276521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.276900 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.276901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.276985 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.277396 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.277400 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.278686 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.280161 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.280168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.280170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.280368 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.280592 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.280593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.280593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.280659 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.280908 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.280914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.280972 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.281073 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.281192 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260928 22:54:03.281826 20490 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.14.189:39427: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee"
member_type: NON_VOTER
last_known_addr {
host: "127.14.14.189"
port: 39427
}
attrs {
promote: true
}
}
dest_uuid: "bca8e77469ae4f95a452cf2b744ddfdf"
cas_config_opid_index: -1
I20260928 22:54:03.282143 20490 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: NON_VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: true } }
I20260928 22:54:03.282605 21051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 75f5bd5a76c94b4bbf6b5e099dfa46ee (127.14.14.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: NON_VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: true } } }
I20260928 22:54:03.282894 20997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:03.283449 21051 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.14.189:39427
I20260928 22:54:03.283594 21050 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: NON_VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: true } } }
I20260928 22:54:03.283664 21050 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:03.283782 20997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 1 FOLLOWER]: Refusing update from remote peer bca8e77469ae4f95a452cf2b744ddfdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 22:54:03.284046 20955 master.cc:537] Master@127.14.14.189:39427 shutting down...
I20260928 22:54:03.284127 21050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: NON_VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 22:54:03.284775 20476 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 22:54:03.285181 20476 consensus_peers.cc:487] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf -> Peer 75f5bd5a76c94b4bbf6b5e099dfa46ee (127.14.14.189:39427): Couldn't send request to peer 75f5bd5a76c94b4bbf6b5e099dfa46ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 22:54:03.286660 20955 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:54:03.286749 20955 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:03.286801 20955 tablet_replica.cc:330] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: stopping tablet replica
I20260928 22:54:03.289206 20955 master.cc:552] Master@127.14.14.189:39427 shutdown complete.
I20260928 22:54:03.289253 20955 master_runner.cc:305] Clearing existing system tablet
I20260928 22:54:03.289387 20955 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:03.289445 20955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:03.290176 20955 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.290652 21055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.290810 20955 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:03.290884 20955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.290913 20955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/wal
uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee"
format_stamp: "Formatted at 2026-09-28 22:54:03 on dist-test-slave-kzfr"
server_key: "457936a9f7b6b151034ccbefb7e0be63"
server_key_iv: "921c09d342d682f12e38fa3feaadf2df"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.291486 20955 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:03.292917 20955 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 22:54:03.293015 20955 log.cc:1196] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 22:54:03.293195 20955 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: Deleting consensus metadata
I20260928 22:54:03.293331 20955 master_runner.cc:315] Copying system tablet from 127.14.14.190:35807
I20260928 22:54:03.293970 20955 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: tablet copy: Beginning tablet copy session from remote peer at address 127.14.14.190:35807
I20260928 22:54:03.300339 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.294322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43774 (local address 127.14.14.190:35807)
0928 22:54:03.294369 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.294371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.294472 (+ 101us) server_negotiation.cc:407] Connection header received
0928 22:54:03.294547 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.294547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.294555 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.294599 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.294937 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.295415 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.296441 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.296616 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.296664 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.297673 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.297674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.297674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.297681 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.299228 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.299645 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.299645 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.299646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.299699 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.299895 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.299895 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.299896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.299994 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.300044 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.300202 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.300269 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:54:03.300391 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.294275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:43774)
0928 22:54:03.294397 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.294410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.294486 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.294619 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.294620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.294811 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.294869 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.294871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.295490 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.295491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.296361 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.296364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.296681 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.296683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.296753 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.297160 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.297165 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.297553 (+ 388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.299333 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.299334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.299335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.299565 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.299766 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.299767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.299767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.299834 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.300053 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.300054 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.300101 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.300171 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.300274 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260928 22:54:03.301059 20511 tablet_copy_service.cc:140] P bca8e77469ae4f95a452cf2b744ddfdf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 75f5bd5a76c94b4bbf6b5e099dfa46ee ({username='oryx', principal='oryx/127.14.14.189@KRBTEST.COM'} at 127.0.0.1:43774)
I20260928 22:54:03.301139 20511 tablet_copy_service.cc:161] P bca8e77469ae4f95a452cf2b744ddfdf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 75f5bd5a76c94b4bbf6b5e099dfa46ee at {username='oryx', principal='oryx/127.14.14.189@KRBTEST.COM'} at 127.0.0.1:43774: session id = 75f5bd5a76c94b4bbf6b5e099dfa46ee-00000000000000000000000000000000
I20260928 22:54:03.301826 20511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 22:54:03.302393 20955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 22:54:03.303330 20955 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: tablet copy: Starting download of 0 data blocks...
I20260928 22:54:03.303480 20955 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: tablet copy: Starting download of 1 WAL segments...
I20260928 22:54:03.304327 20955 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 22:54:03.305300 20511 tablet_copy_service.cc:342] P bca8e77469ae4f95a452cf2b744ddfdf: Request end of tablet copy session 75f5bd5a76c94b4bbf6b5e099dfa46ee-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.14.189@KRBTEST.COM'} at 127.0.0.1:43774
I20260928 22:54:03.305357 20511 tablet_copy_service.cc:433] P bca8e77469ae4f95a452cf2b744ddfdf: ending tablet copy session 75f5bd5a76c94b4bbf6b5e099dfa46ee-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 75f5bd5a76c94b4bbf6b5e099dfa46ee
I20260928 22:54:03.307919 20955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:03.309142 21062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:03.309201 21063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:03.310747 21065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:03.310786 20955 server_base.cc:1034] running on GCE node
I20260928 22:54:03.310933 20955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:03.311074 20955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:03.312206 20955 hybrid_clock.cc:648] HybridClock initialized: now 1790636043312199 us; error 36 us; skew 500 ppm
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:03.313496 20955 init.cc:377] Logged in from keytab as oryx/127.14.14.189@KRBTEST.COM (short username oryx)
I20260928 22:54:03.313830 20955 webserver.cc:460] Webserver started at http://127.14.14.189:40779/ using document root <none> and password file <none>
I20260928 22:54:03.313966 20955 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:03.314011 20955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:03.314646 20955 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:54:03.316200 21072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.316336 20955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:54:03.316387 20955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/wal
uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee"
format_stamp: "Formatted at 2026-09-28 22:54:03 on dist-test-slave-kzfr"
server_key: "457936a9f7b6b151034ccbefb7e0be63"
server_key_iv: "921c09d342d682f12e38fa3feaadf2df"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.316454 20955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:03.325845 20955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:03.327543 20955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:03.330803 20955 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.189:39427
I20260928 22:54:03.331091 21124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.189:39427 every 8 connection(s)
I20260928 22:54:03.331195 20955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-1/data/info.pb
I20260928 22:54:03.332077 21125 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:54:03.332567 21125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: Bootstrap starting.
I20260928 22:54:03.335417 21125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:54:03.335654 21125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: Bootstrap complete.
I20260928 22:54:03.335842 21125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: NON_VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: true } }
I20260928 22:54:03.335927 21125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 75f5bd5a76c94b4bbf6b5e099dfa46ee, State: Initialized, Role: LEARNER
I20260928 22:54:03.335958 21125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: NON_VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: true } }
I20260928 22:54:03.336117 21125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:54:03.336122 21128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: NON_VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: true } } }
I20260928 22:54:03.336256 21128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: This master's current role is: LEARNER
I20260928 22:54:03.337544 21139 catalog_manager.cc:1259] Loaded cluster ID: f7c612814c4549cf84628db94f6e15d2
I20260928 22:54:03.337618 21139 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: loading cluster ID for follower catalog manager: success
I20260928 22:54:03.338577 21139 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: acquiring CA information for follower catalog manager: success
I20260928 22:54:03.338850 21139 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:54:03.361630 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.355122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.14.14.189:39427)
0928 22:54:03.355261 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.355262 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.355269 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:54:03.355292 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.355293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.355301 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.355373 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.355912 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.356601 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.357326 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.357540 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.357653 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.358905 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.358907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.358907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.358922 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.360382 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.360781 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.360782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.360782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.360840 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.361039 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.361041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.361041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.361149 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.361251 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.361372 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.361461 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for oryx/127.14.14.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.14.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.14.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.188.keytab.
Entry for principal oryx/127.14.14.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.188.keytab.
I20260928 22:54:03.389050 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.188:39071
--webserver_interface=127.14.14.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.188.keytab
--principal=oryx/127.14.14.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:35807,127.14.14.189:39427,127.14.14.188:39071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:03.470599 21143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:03.470811 21143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:03.470834 21143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:03.472002 21143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:54:03.472046 21143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:03.472060 21143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:03.472074 21143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:54:03.472127 21143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:54:03.473661 21143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:35807,127.14.14.189:39427,127.14.14.188:39071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.188.keytab
--principal=oryx/127.14.14.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.188:39071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.14.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:03.473923 21143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:03.474187 21143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:03.479236 21148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:03.479270 21149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:03.479344 21143 server_base.cc:1034] running on GCE node
W20260928 22:54:03.479290 21151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:03.479668 21143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:03.479984 21143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:03.481170 21143 hybrid_clock.cc:648] HybridClock initialized: now 1790636043481158 us; error 47 us; skew 500 ppm
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:03.493490 21143 init.cc:377] Logged in from keytab as oryx/127.14.14.188@KRBTEST.COM (short username oryx)
I20260928 22:54:03.494053 21143 webserver.cc:460] Webserver started at http://127.14.14.188:34375/ using document root <none> and password file <none>
I20260928 22:54:03.494300 21143 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:03.494369 21143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:03.494529 21143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:03.495632 21143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/data/instance:
uuid: "43f4101c5f22495b9950e8779f0fa717"
format_stamp: "Formatted at 2026-09-28 22:54:03 on dist-test-slave-kzfr"
server_key: "f0e785a2099501ce1d9336aafb1cb6ce"
server_key_iv: "88edb2d4b86fb1c9c8bcd86729cece4b"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.496001 21143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/wal/instance:
uuid: "43f4101c5f22495b9950e8779f0fa717"
format_stamp: "Formatted at 2026-09-28 22:54:03 on dist-test-slave-kzfr"
server_key: "f0e785a2099501ce1d9336aafb1cb6ce"
server_key_iv: "88edb2d4b86fb1c9c8bcd86729cece4b"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.497352 21143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:54:03.498180 21158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.498364 21143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.498479 21143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/wal
uuid: "43f4101c5f22495b9950e8779f0fa717"
format_stamp: "Formatted at 2026-09-28 22:54:03 on dist-test-slave-kzfr"
server_key: "f0e785a2099501ce1d9336aafb1cb6ce"
server_key_iv: "88edb2d4b86fb1c9c8bcd86729cece4b"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.498553 21143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:03.515290 21143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:03.517036 21143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:03.517194 21143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:03.521440 21143 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.188:39071
I20260928 22:54:03.521476 21210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.188:39071 every 8 connection(s)
I20260928 22:54:03.521874 21143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/data/info.pb
I20260928 22:54:03.522854 21211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:03.524448 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 21143
I20260928 22:54:03.524302 21211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.188@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.14.188@KRBTEST.COM
I20260928 22:54:03.534071 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.525255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.14.14.190:35807)
0928 22:54:03.525300 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.525302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.525472 (+ 170us) server_negotiation.cc:407] Connection header received
0928 22:54:03.526080 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.526082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.526094 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.526146 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.526864 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.527638 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.528518 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.528822 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.528878 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.531184 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.531185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.531185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.531192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.532578 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.533031 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.533032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.533033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.533095 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.533390 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.533391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.533391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.533528 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.533627 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.533897 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.533981 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:54:03.534468 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.525057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:43784)
0928 22:54:03.525333 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.525377 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.525949 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.526224 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.526231 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.526561 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.526764 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.526770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.527748 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.527749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.528434 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.528436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.528923 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.528924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.529051 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.529684 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.529692 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.531101 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.532665 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.532672 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.532674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.532922 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.533164 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.533165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.533165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.533323 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.533623 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.533630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.533767 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.533877 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.534036 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":77,"threads_started":1}
I20260928 22:54:03.535013 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.525235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44872 (local address 127.14.14.188:39071)
0928 22:54:03.525358 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.525369 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.525400 (+ 31us) server_negotiation.cc:407] Connection header received
0928 22:54:03.525952 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.525962 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.526036 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.526126 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.526737 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.527951 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.528763 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.529164 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.529236 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.531810 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.531817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.531817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.531823 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.533545 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.533925 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.533928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.533928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.533996 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.534214 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.534215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.534215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.534419 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.534587 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.534822 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.534897 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":49,"threads_started":1}
I20260928 22:54:03.535012 21211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.188@KRBTEST.COM for oryx/127.14.14.189@KRBTEST.COM
I20260928 22:54:03.541177 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.535338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:39427 (local address 127.0.0.1:52180)
0928 22:54:03.535371 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.535400 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.535443 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.535614 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.535615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.535736 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.535761 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.535762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.536576 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.536576 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.537172 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.537173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.537505 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.537506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.537569 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.537926 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.537930 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.538972 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.540476 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.540476 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.540477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.540614 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.540762 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.540763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.540763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.540807 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.540983 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.540983 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.541014 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.541061 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.541121 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 22:54:03.541196 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.535402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.14.14.189:39427)
0928 22:54:03.535471 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.535472 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.535481 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:54:03.535508 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.535509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.535521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.535569 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.535878 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.536511 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.537242 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.537436 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.537491 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.539046 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.539047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.539048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.539054 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.540407 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.540665 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.540665 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.540665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.540707 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.540847 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.540847 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.540848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.540933 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.540989 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.541078 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.541138 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:54:03.541735 21211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.188@KRBTEST.COM for oryx/127.14.14.188@KRBTEST.COM
I20260928 22:54:03.547114 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.541999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39071 (local address 127.0.0.1:44874)
0928 22:54:03.542047 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.542057 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.542095 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.542200 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.542201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.542302 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.542326 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.542326 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.542875 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.542875 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.543325 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.543326 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.543638 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.543639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.543674 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.544066 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.544074 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.545071 (+ 997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.546372 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.546373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.546373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.546528 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.546691 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.546692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.546692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.546740 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.546901 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.546902 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.546932 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.546991 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.547058 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 22:54:03.547173 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.542056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.14.14.188:39071)
0928 22:54:03.542104 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.542106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.542121 (+ 15us) server_negotiation.cc:407] Connection header received
0928 22:54:03.542140 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.542140 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.542152 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.542199 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.542379 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.542802 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.543391 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.543577 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.543624 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.545142 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.545143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.545143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.545149 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.546306 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.546591 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.546592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.546592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.546635 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.546777 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.546778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.546778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.546855 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.546896 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.547007 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.547099 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":31}
I20260928 22:54:03.548471 21211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: Bootstrap starting.
I20260928 22:54:03.549297 21211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:03.549645 21211 log.cc:824] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:03.550348 21211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: No bootstrap required, opened a new log
I20260928 22:54:03.551760 21211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } }
I20260928 22:54:03.551879 21211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:03.551927 21211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f4101c5f22495b9950e8779f0fa717, State: Initialized, Role: FOLLOWER
I20260928 22:54:03.552055 21211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } }
I20260928 22:54:03.552356 21211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:54:03.552317 21216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } } }
I20260928 22:54:03.552426 21216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:54:03.554046 21227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:54:03.554100 21227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:54:03.556353 21143 master_runner.cc:428] Detected that this master 43f4101c5f22495b9950e8779f0fa717 is joining an existing cluster
I20260928 22:54:03.556404 21143 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.14.188:39071
I20260928 22:54:03.565266 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.557857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:39427 (local address 127.0.0.1:52190)
0928 22:54:03.558505 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.558514 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.558592 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.558758 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.558759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.558996 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.559071 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.559073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.560104 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.560105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.561225 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.561226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.561767 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.561767 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.561835 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.562233 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.562236 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.562526 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.564255 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.564256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.564256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.564447 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.564686 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.564686 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.564686 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.564738 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.565017 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.565018 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.565056 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.565116 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.565191 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":186,"thread_start_us":113,"threads_started":1}
I20260928 22:54:03.565335 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.557913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.14.14.189:39427)
0928 22:54:03.557954 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.557955 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.558560 (+ 605us) server_negotiation.cc:407] Connection header received
0928 22:54:03.558642 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.558643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.558656 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.558710 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.559159 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.559999 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.561330 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.561651 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.561688 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.562615 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.562616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.562617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.562624 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.564168 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.564529 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.564530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.564530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.564609 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.564795 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.564795 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.564796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.564932 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.565005 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.565162 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.565241 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:54:03.585119 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.558021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.14.14.188:39071)
0928 22:54:03.558057 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.558059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.558407 (+ 348us) server_negotiation.cc:407] Connection header received
0928 22:54:03.558466 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.558467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.558477 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.558536 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.558912 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.559630 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.560654 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.560943 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.560983 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.582112 (+ 21129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.582114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.582115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.582124 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.583838 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.584228 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.584230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.584230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.584288 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.584497 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.584498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.584499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.584634 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.584712 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.584876 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.585028 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":16}
I20260928 22:54:03.585120 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.557855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39071 (local address 127.0.0.1:44890)
0928 22:54:03.558336 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.558345 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.558418 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.558557 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.558558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.558745 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.558817 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.558818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.559719 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.559720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.560563 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.560566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.561063 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.561063 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.561119 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.561622 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.561627 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.581965 (+ 20338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.583924 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.583925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.583926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.584148 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.584364 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.584365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.584365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.584433 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.584704 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.584705 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.584758 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.584833 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.584970 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"spinlock_wait_cycles":9472,"thread_start_us":109,"threads_started":1}
I20260928 22:54:03.585704 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.558220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.14.14.190:35807)
0928 22:54:03.558283 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.558285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.558739 (+ 454us) server_negotiation.cc:407] Connection header received
0928 22:54:03.558834 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.558835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.558846 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.558877 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.559271 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.559999 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.561330 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.561647 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.561688 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.582729 (+ 21041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.582731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.582732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.582741 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.584464 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.584789 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.584791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.584791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.584852 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.585080 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.585081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.585081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.585218 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.585259 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.585520 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.585619 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:54:03.585878 21234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.558146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:43800)
0928 22:54:03.558687 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.558696 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.558761 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.558936 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.558936 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.559140 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.559191 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.559193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.560104 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.560105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.561218 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.561220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.561764 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.561765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.561830 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.582207 (+ 20377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.582213 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.582615 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.584531 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.584531 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.584532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.584725 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.584935 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.584936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.584936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.584992 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.585345 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.585346 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.585409 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.585481 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.585613 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":34,"thread_start_us":96,"threads_started":1}
I20260928 22:54:03.586264 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 20553
I20260928 22:54:03.591816 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:45887
--local_ip_for_outbound_sockets=127.14.14.129
--tserver_master_addrs=127.14.14.190:35807
--webserver_port=35021
--webserver_interface=127.14.14.129
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.129.keytab
--principal=oryx/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.14.190:35807,127.14.14.189:39427
--tserver_master_addrs=127.14.14.190:35807,127.14.14.189:39427,127.14.14.188:39071 with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636042, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.190@KRBTEST.COM for oryx/127.14.14.188@KRBTEST.COM
I20260928 22:54:03.601230 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.593271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39071 (local address 127.0.0.1:44906)
0928 22:54:03.593326 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.593336 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.593371 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.593535 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.593536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.593729 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.593781 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.593784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.594452 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.594453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.595300 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.595303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.595802 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.595803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.595878 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.596350 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.596355 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.598085 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.600046 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.600047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.600047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.600291 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.600527 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.600528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.600528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.600603 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.600898 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.600900 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.600953 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.601019 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.601124 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:54:03.601908 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.593330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.14.14.188:39071)
0928 22:54:03.593370 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.593372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.593382 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:54:03.593442 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.593443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.593462 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.593518 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.593863 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.594372 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.595394 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.595719 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.595804 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.598161 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.598162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.598163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.598170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.599951 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.600374 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.600375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.600375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.600435 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.600669 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.600669 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.600670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.600806 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.600867 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.601675 (+ 808us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.601758 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:54:03.602420 20490 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.14.188:39071: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "43f4101c5f22495b9950e8779f0fa717"
member_type: NON_VOTER
last_known_addr {
host: "127.14.14.188"
port: 39071
}
attrs {
promote: true
}
}
dest_uuid: "bca8e77469ae4f95a452cf2b744ddfdf"
cas_config_opid_index: 5
I20260928 22:54:03.602662 20490 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: NON_VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: true } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } }
I20260928 22:54:03.603016 21050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 43f4101c5f22495b9950e8779f0fa717 (127.14.14.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: NON_VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: true } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } } }
I20260928 22:54:03.603260 21185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:03.604024 21050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.14.188:39071
I20260928 22:54:03.604099 21050 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: NON_VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: true } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } } }
I20260928 22:54:03.604156 21050 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:03.604247 21185 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 1 FOLLOWER]: Refusing update from remote peer bca8e77469ae4f95a452cf2b744ddfdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 22:54:03.604408 21143 master.cc:537] Master@127.14.14.188:39071 shutting down...
I20260928 22:54:03.604501 21051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 22:54:03.605947 20477 consensus_peers.cc:487] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf -> Peer 43f4101c5f22495b9950e8779f0fa717 (127.14.14.188:39071): Couldn't send request to peer 43f4101c5f22495b9950e8779f0fa717. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 22:54:03.607676 21143 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:54:03.607769 21143 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:03.607817 21143 tablet_replica.cc:330] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: stopping tablet replica
I20260928 22:54:03.607996 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.603173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.14.14.189:39427)
0928 22:54:03.603333 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.603335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.603525 (+ 190us) server_negotiation.cc:407] Connection header received
0928 22:54:03.603551 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.603552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.603576 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.603634 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:03.603797 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.604514 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.606398 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.607046 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.607118 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.607192 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.607889 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.607932 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260928 22:54:03.608233 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.603073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:39427 (local address 127.0.0.1:52198)
0928 22:54:03.603395 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.603407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.603454 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.603690 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.603690 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.603692 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:03.603729 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.603730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.604730 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.604731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.606300 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.606303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.607487 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.607488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.607555 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.607577 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.607632 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.607667 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307}
I20260928 22:54:03.608596 21099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 1 LEARNER]: Refusing update from remote peer bca8e77469ae4f95a452cf2b744ddfdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 22:54:03.608769 21051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: NON_VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 22:54:03.609320 21099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 43f4101c5f22495b9950e8779f0fa717 (127.14.14.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: NON_VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: true } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } } }
I20260928 22:54:03.610725 21143 master.cc:552] Master@127.14.14.188:39071 shutdown complete.
I20260928 22:54:03.610778 21143 master_runner.cc:305] Clearing existing system tablet
I20260928 22:54:03.610919 21143 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:03.610984 21143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:03.611083 21238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: NON_VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: true } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } } }
I20260928 22:54:03.611189 21238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: This master's current role is: LEARNER
I20260928 22:54:03.611320 21128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca8e77469ae4f95a452cf2b744ddfdf. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: NON_VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: true } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } } }
I20260928 22:54:03.611433 21128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: This master's current role is: LEARNER
I20260928 22:54:03.611716 21143 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:54:03.612286 21240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.612475 21143 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:03.612548 21143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:54:03.612577 21143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/wal
uuid: "43f4101c5f22495b9950e8779f0fa717"
format_stamp: "Formatted at 2026-09-28 22:54:03 on dist-test-slave-kzfr"
server_key: "f0e785a2099501ce1d9336aafb1cb6ce"
server_key_iv: "88edb2d4b86fb1c9c8bcd86729cece4b"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.613494 21143 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:03.613903 21051 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf: attempting to promote NON_VOTER 75f5bd5a76c94b4bbf6b5e099dfa46ee to VOTER
I20260928 22:54:03.614107 21051 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } }
I20260928 22:54:03.614650 21099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 1 LEARNER]: Refusing update from remote peer bca8e77469ae4f95a452cf2b744ddfdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 22:54:03.614678 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.614565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39071 (local address 127.0.0.1:44912)
0928 22:54:03.614611 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.614651 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 22:54:03.614831 21050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 22:54:03.615028 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.614924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39071 (local address 127.0.0.1:44924)
0928 22:54:03.614978 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.615003 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 22:54:03.615096 20477 consensus_peers.cc:487] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf -> Peer 43f4101c5f22495b9950e8779f0fa717 (127.14.14.188:39071): Couldn't send request to peer 43f4101c5f22495b9950e8779f0fa717. Status: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 22:54:03.615419 21143 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 22:54:03.615492 21143 log.cc:1196] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 22:54:03.615427 21050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 75f5bd5a76c94b4bbf6b5e099dfa46ee (127.14.14.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } } }
I20260928 22:54:03.615701 21143 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: Deleting consensus metadata
I20260928 22:54:03.615935 21143 master_runner.cc:315] Copying system tablet from 127.14.14.190:35807
I20260928 22:54:03.616305 21050 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } } }
I20260928 22:54:03.616384 21050 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:03.616436 21050 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } } }
I20260928 22:54:03.616494 21050 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:03.616691 21143 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: tablet copy: Beginning tablet copy session from remote peer at address 127.14.14.190:35807
I20260928 22:54:03.616751 21099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 75f5bd5a76c94b4bbf6b5e099dfa46ee (127.14.14.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } } }
I20260928 22:54:03.617633 21128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } } }
I20260928 22:54:03.617722 21128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:54:03.625136 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.618161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.14.14.190:35807)
0928 22:54:03.618204 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.618207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.618416 (+ 209us) server_negotiation.cc:407] Connection header received
0928 22:54:03.618500 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.618502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.618512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.618566 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.618911 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.619662 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.620794 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.621088 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.621162 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.622294 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.622295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.622296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.622302 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.623915 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.624323 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.624324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.624325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.624388 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.624597 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.624597 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.624598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.624732 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.624802 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.624962 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.625042 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:54:03.625155 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.618220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:43816)
0928 22:54:03.618353 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.618363 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.618442 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.618579 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.618579 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.618773 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.618828 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.618830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.619748 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.619749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.620559 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.620561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.621173 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.621173 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.621242 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.621786 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.621791 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.622196 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.624007 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.624008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.624009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.624240 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.624461 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.624461 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.624462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.624528 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.624802 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.624803 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.624851 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.624920 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.625034 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260928 22:54:03.625720 20511 tablet_copy_service.cc:140] P bca8e77469ae4f95a452cf2b744ddfdf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 43f4101c5f22495b9950e8779f0fa717 ({username='oryx', principal='oryx/127.14.14.188@KRBTEST.COM'} at 127.0.0.1:43816)
I20260928 22:54:03.625779 20511 tablet_copy_service.cc:161] P bca8e77469ae4f95a452cf2b744ddfdf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 43f4101c5f22495b9950e8779f0fa717 at {username='oryx', principal='oryx/127.14.14.188@KRBTEST.COM'} at 127.0.0.1:43816: session id = 43f4101c5f22495b9950e8779f0fa717-00000000000000000000000000000000
I20260928 22:54:03.626368 20511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 22:54:03.626894 21143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 22:54:03.628207 21143 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: tablet copy: Starting download of 0 data blocks...
I20260928 22:54:03.628346 21143 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: tablet copy: Starting download of 1 WAL segments...
I20260928 22:54:03.629196 21143 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 22:54:03.630177 20511 tablet_copy_service.cc:342] P bca8e77469ae4f95a452cf2b744ddfdf: Request end of tablet copy session 43f4101c5f22495b9950e8779f0fa717-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.14.188@KRBTEST.COM'} at 127.0.0.1:43816
I20260928 22:54:03.630230 20511 tablet_copy_service.cc:433] P bca8e77469ae4f95a452cf2b744ddfdf: ending tablet copy session 43f4101c5f22495b9950e8779f0fa717-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 43f4101c5f22495b9950e8779f0fa717
I20260928 22:54:03.632593 21143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:03.633872 21248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:03.633924 21249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:03.634123 21251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:03.636265 21143 server_base.cc:1034] running on GCE node
I20260928 22:54:03.636372 21143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:03.636518 21143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:03.637635 21143 hybrid_clock.cc:648] HybridClock initialized: now 1790636043637621 us; error 33 us; skew 500 ppm
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:03.639143 21143 init.cc:377] Logged in from keytab as oryx/127.14.14.188@KRBTEST.COM (short username oryx)
I20260928 22:54:03.639398 21143 webserver.cc:460] Webserver started at http://127.14.14.188:38625/ using document root <none> and password file <none>
I20260928 22:54:03.639533 21143 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:03.639575 21143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:03.640318 21143 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:03.640920 21258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.641095 21143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.641147 21143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/wal
uuid: "43f4101c5f22495b9950e8779f0fa717"
format_stamp: "Formatted at 2026-09-28 22:54:03 on dist-test-slave-kzfr"
server_key: "f0e785a2099501ce1d9336aafb1cb6ce"
server_key_iv: "88edb2d4b86fb1c9c8bcd86729cece4b"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.641204 21143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:03.652451 21143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:03.654317 21143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:03.657341 21143 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.188:39071
I20260928 22:54:03.657722 21143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/master-2/data/info.pb
I20260928 22:54:03.657990 21310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.188:39071 every 8 connection(s)
I20260928 22:54:03.658694 21311 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:54:03.659171 21311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: Bootstrap starting.
I20260928 22:54:03.662096 21311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:54:03.662340 21311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: Bootstrap complete.
I20260928 22:54:03.662532 21311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } }
I20260928 22:54:03.662619 21311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 43f4101c5f22495b9950e8779f0fa717, State: Initialized, Role: LEARNER
I20260928 22:54:03.662653 21311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } }
I20260928 22:54:03.662807 21311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:54:03.664248 21325 catalog_manager.cc:1259] Loaded cluster ID: f7c612814c4549cf84628db94f6e15d2
I20260928 22:54:03.664347 21325 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: loading cluster ID for follower catalog manager: success
I20260928 22:54:03.665196 21314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } } }
I20260928 22:54:03.665288 21314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [sys.catalog]: This master's current role is: LEARNER
I20260928 22:54:03.665658 21325 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: acquiring CA information for follower catalog manager: success
I20260928 22:54:03.665994 21325 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 22:54:03.694870 21236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:03.695062 21236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:03.695086 21236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:03.696214 21236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:03.696255 21236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:03.696281 21236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:54:03.697804 21236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.129.keytab
--principal=oryx/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:45887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=35021
--tserver_master_addrs=127.14.14.190:35807,127.14.14.189:39427,127.14.14.188:39071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:03.698031 21236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:03.698246 21236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:54:03.704103 21236 server_base.cc:1034] running on GCE node
W20260928 22:54:03.704111 21331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:03.704224 21333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:03.704283 21330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:03.704532 21236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:03.704797 21236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:03.705943 21236 hybrid_clock.cc:648] HybridClock initialized: now 1790636043705928 us; error 31 us; skew 500 ppm
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:03.708400 21236 init.cc:377] Logged in from keytab as oryx/127.14.14.129@KRBTEST.COM (short username oryx)
I20260928 22:54:03.708954 21236 webserver.cc:460] Webserver started at http://127.14.14.129:35021/ using document root <none> and password file <none>
I20260928 22:54:03.709213 21236 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:03.709285 21236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:03.710908 21236 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:03.711541 21340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.711683 21236 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:03.711766 21236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "ae618bb3e59841e0a60be96d1b74235e"
format_stamp: "Formatted at 2026-09-28 22:54:02 on dist-test-slave-kzfr"
server_key: "a4395805f290b68a4b46ab24606a4058"
server_key_iv: "4d6f720e8b4979738c5621d686f24049"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.712105 21236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:03.726724 21236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:03.728586 21236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:03.728741 21236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:03.728966 21236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:03.729281 21236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:03.729315 21236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.729337 21236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:03.729383 21236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.735694 21236 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:45887
I20260928 22:54:03.735780 21453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:45887 every 8 connection(s)
I20260928 22:54:03.736133 21236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:54:03.737571 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 21236
I20260928 22:54:03.737742 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 20687
I20260928 22:54:03.743362 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:42869
--local_ip_for_outbound_sockets=127.14.14.130
--tserver_master_addrs=127.14.14.190:35807
--webserver_port=35967
--webserver_interface=127.14.14.130
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.130.keytab
--principal=oryx/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.14.190:35807,127.14.14.189:39427
--tserver_master_addrs=127.14.14.190:35807,127.14.14.189:39427,127.14.14.188:39071 with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.189@KRBTEST.COM
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.188@KRBTEST.COM
I20260928 22:54:03.747892 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.736967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:43245 (local address 127.14.14.189:39427)
0928 22:54:03.737030 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.737032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.737252 (+ 220us) server_negotiation.cc:407] Connection header received
0928 22:54:03.738710 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.738711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.738725 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.738755 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.739527 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.740027 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.741105 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.741353 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.741414 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.744236 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.744237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.744238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.744245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.746111 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.746661 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.746663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.746663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.746722 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.747084 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.747086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.747086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.747234 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.747282 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.747681 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.747764 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:54:03.748698 21460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.736882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:39427 (local address 127.14.14.129:43245)
0928 22:54:03.737129 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.737171 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.738599 (+ 1428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.738904 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.738909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.739245 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.739428 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.739432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.740121 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.740122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.741036 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.741038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.741419 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.741420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.741564 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.742129 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.742140 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.744144 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.746200 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.746208 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.746210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.746569 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.746856 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.746857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.746857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.746998 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.747333 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.747340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.747522 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.748228 (+ 706us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.748442 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":96,"threads_started":1}
I20260928 22:54:03.749502 21456 heartbeater.cc:347] Connected to a master server at 127.14.14.189:39427
I20260928 22:54:03.749639 21456 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:03.749857 21456 heartbeater.cc:511] Master 127.14.14.189:39427 requested a full tablet report, sending...
I20260928 22:54:03.749966 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.736884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.129:58179)
0928 22:54:03.737303 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.737312 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.741400 (+ 4088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.741622 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.741623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.742209 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.742263 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.742264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.743199 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.743200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.744048 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.744050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.744878 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.744879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.744939 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.745560 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.745565 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.747022 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.748823 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.748824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.748824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.749014 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.749214 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.749214 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.749215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.749273 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.749628 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.749629 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.749668 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.749731 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.749830 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":44,"thread_start_us":75,"threads_started":1}
I20260928 22:54:03.750233 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.737137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:58179 (local address 127.14.14.190:35807)
0928 22:54:03.737220 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.737222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.737382 (+ 160us) server_negotiation.cc:407] Connection header received
0928 22:54:03.741490 (+ 4108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.741491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.741505 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.741558 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.742367 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.743096 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.744151 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.744436 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.744792 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.747114 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.747116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.747116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.747123 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.748724 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.749083 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.749084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.749084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.749137 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.749348 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.749349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.749350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.749493 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.749544 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.749774 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.749848 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 22:54:03.750419 21089 ts_manager.cc:194] Registered new tserver with Master: ae618bb3e59841e0a60be96d1b74235e (127.14.14.129:45887)
I20260928 22:54:03.750738 21455 heartbeater.cc:347] Connected to a master server at 127.14.14.190:35807
I20260928 22:54:03.750782 21455 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:03.750844 21455 heartbeater.cc:511] Master 127.14.14.190:35807 requested a full tablet report, sending...
I20260928 22:54:03.751094 20490 ts_manager.cc:194] Re-registered known tserver with Master: ae618bb3e59841e0a60be96d1b74235e (127.14.14.129:45887)
I20260928 22:54:03.751472 20490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:58179
I20260928 22:54:03.752995 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.736885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39071 (local address 127.14.14.129:54039)
0928 22:54:03.737216 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.737228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.738587 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.741716 (+ 3129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.741717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.741914 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.742023 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.742025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.744266 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.744267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.745175 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.745178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.745764 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.745765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.745839 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.746259 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.746263 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.748438 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.752050 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.752050 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.752051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.752201 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.752424 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.752424 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.752425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.752469 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.752770 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.752770 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.752804 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.752852 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.752929 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260928 22:54:03.753157 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.737179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:54039 (local address 127.14.14.188:39071)
0928 22:54:03.741535 (+ 4356us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.741537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.741544 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:54:03.741570 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.741570 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.741614 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.741668 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.743154 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.744168 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.745275 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.745643 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.745707 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.748545 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.748547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.748547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.748556 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.751654 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.752284 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.752285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.752286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.752357 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.752533 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.752533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.752534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.752695 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.752761 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.752905 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.753015 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4331,"thread_start_us":60,"threads_started":1}
I20260928 22:54:03.753521 21454 heartbeater.cc:347] Connected to a master server at 127.14.14.188:39071
I20260928 22:54:03.753571 21454 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:03.753672 21454 heartbeater.cc:511] Master 127.14.14.188:39071 requested a full tablet report, sending...
I20260928 22:54:03.753993 21275 ts_manager.cc:194] Registered new tserver with Master: ae618bb3e59841e0a60be96d1b74235e (127.14.14.129:45887)
W20260928 22:54:03.831744 21462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:03.831938 21462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:03.831962 21462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:03.833106 21462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:03.833148 21462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:03.833200 21462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:54:03.834960 21462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.130.keytab
--principal=oryx/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:42869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=35967
--tserver_master_addrs=127.14.14.190:35807,127.14.14.189:39427,127.14.14.188:39071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:03.835194 21462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:03.835449 21462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:03.840766 21468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:03.840889 21462 server_base.cc:1034] running on GCE node
W20260928 22:54:03.840788 21470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:03.840782 21467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:03.841195 21462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:03.841455 21462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:03.842617 21462 hybrid_clock.cc:648] HybridClock initialized: now 1790636043842607 us; error 47 us; skew 500 ppm
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:03.844956 21462 init.cc:377] Logged in from keytab as oryx/127.14.14.130@KRBTEST.COM (short username oryx)
I20260928 22:54:03.845418 21462 webserver.cc:460] Webserver started at http://127.14.14.130:35967/ using document root <none> and password file <none>
I20260928 22:54:03.845717 21462 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:03.845800 21462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:03.847110 21462 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:54:03.847676 21477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.847847 21462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:54:03.847941 21462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "c89c8d94bf79445189e99922cdefdef8"
format_stamp: "Formatted at 2026-09-28 22:54:02 on dist-test-slave-kzfr"
server_key: "f934d7809d688736af00b61894e7d679"
server_key_iv: "032e2c182583ec233c49c41e955b2bf4"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.848284 21462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:03.863128 21462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:03.865056 21462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:03.865211 21462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:03.865437 21462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:03.865759 21462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:03.865816 21462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.865855 21462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:03.865885 21462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.871532 21462 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:42869
I20260928 22:54:03.871579 21590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:42869 every 8 connection(s)
I20260928 22:54:03.871898 21462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.188@KRBTEST.COM
I20260928 22:54:03.879799 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 21462
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.189@KRBTEST.COM
I20260928 22:54:03.879976 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 20821
I20260928 22:54:03.883390 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.872930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:55845 (local address 127.14.14.188:39071)
0928 22:54:03.873004 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.873006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.873084 (+ 78us) server_negotiation.cc:407] Connection header received
0928 22:54:03.873997 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.873998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.874012 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.874043 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.874734 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.875490 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.876575 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.876817 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.876892 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.879633 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.879634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.879634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.879642 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.881763 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.882322 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.882323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.882324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.882380 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.882676 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.882677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.882677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.882804 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.882854 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.883217 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.883297 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 22:54:03.883390 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.872870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:52985 (local address 127.14.14.189:39427)
0928 22:54:03.872994 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.872997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.873007 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:54:03.873988 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.873989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.874002 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.874037 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.874733 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.875497 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.876610 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.876954 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.877537 (+ 583us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.880264 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.880265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.880266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.880272 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.881882 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.882322 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.882323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.882324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.882379 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.882676 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.882677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.882677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.882804 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.882848 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.883217 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.883296 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 22:54:03.883431 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.872814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:49275 (local address 127.14.14.190:35807)
0928 22:54:03.872858 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.872860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.872870 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:54:03.873975 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.873976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.873990 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.874054 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:03.874726 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.875230 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.876333 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.876581 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.876646 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.879122 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:03.879123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:03.879124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:03.879130 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:03.882010 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.882495 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.882496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.882497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.882555 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:03.882739 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:03.882740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:03.882740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:03.882893 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:03.882940 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.883206 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.883290 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:54:03.883937 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.872523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:39427 (local address 127.14.14.130:52985)
0928 22:54:03.872874 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.872884 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.873885 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.874144 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.874144 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.874474 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.874626 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.874631 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.875590 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.875590 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.876503 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.876505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.877552 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.877553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.877657 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.878060 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.878066 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.880177 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.881970 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.881970 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.881971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.882233 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.882456 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.882457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.882457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.882595 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.882899 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.882906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.883057 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.883541 (+ 484us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.883703 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":54,"thread_start_us":89,"threads_started":1}
I20260928 22:54:03.883937 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.872523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39071 (local address 127.14.14.130:55845)
0928 22:54:03.873014 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.873023 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.873885 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.874144 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.874144 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.874474 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.874626 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.874628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.875590 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.875590 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.876495 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.876497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.876892 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.876893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.876958 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.877510 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.877514 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.879522 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.881856 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.881863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.881865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.882237 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.882466 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.882467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.882467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.882595 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.882899 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.882906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.883057 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.883490 (+ 433us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.883748 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":158,"spinlock_wait_cycles":7552,"thread_start_us":97,"threads_started":1}
I20260928 22:54:03.884528 21591 heartbeater.cc:347] Connected to a master server at 127.14.14.188:39071
I20260928 22:54:03.884658 21591 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:03.884689 21593 heartbeater.cc:347] Connected to a master server at 127.14.14.189:39427
I20260928 22:54:03.884732 21593 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:03.884845 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.872522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.130:49275)
0928 22:54:03.872703 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.872748 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.873861 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.874131 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.874134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.874438 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:03.874624 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.874628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.875338 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.875339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.876244 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.876246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.876656 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.876657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.876799 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.877358 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:03.877369 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:03.879027 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:03.882103 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.882104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.882105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.882409 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.882619 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:03.882620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:03.882620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:03.882678 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:03.882989 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:03.882989 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:03.883061 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.884003 (+ 942us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.884458 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260928 22:54:03.884862 21593 heartbeater.cc:511] Master 127.14.14.189:39427 requested a full tablet report, sending...
I20260928 22:54:03.884862 21591 heartbeater.cc:511] Master 127.14.14.188:39071 requested a full tablet report, sending...
I20260928 22:54:03.885511 21275 ts_manager.cc:194] Registered new tserver with Master: c89c8d94bf79445189e99922cdefdef8 (127.14.14.130:42869)
I20260928 22:54:03.885726 21089 ts_manager.cc:194] Registered new tserver with Master: c89c8d94bf79445189e99922cdefdef8 (127.14.14.130:42869)
I20260928 22:54:03.886148 21592 heartbeater.cc:347] Connected to a master server at 127.14.14.190:35807
I20260928 22:54:03.886189 21592 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:03.886265 21592 heartbeater.cc:511] Master 127.14.14.190:35807 requested a full tablet report, sending...
I20260928 22:54:03.886502 20490 ts_manager.cc:194] Re-registered known tserver with Master: c89c8d94bf79445189e99922cdefdef8 (127.14.14.130:42869)
I20260928 22:54:03.886826 20490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:49275
I20260928 22:54:03.887781 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:36527
--local_ip_for_outbound_sockets=127.14.14.131
--tserver_master_addrs=127.14.14.190:35807
--webserver_port=34151
--webserver_interface=127.14.14.131
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.131.keytab
--principal=oryx/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.14.190:35807,127.14.14.189:39427
--tserver_master_addrs=127.14.14.190:35807,127.14.14.189:39427,127.14.14.188:39071 with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:03.968457 21598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:03.968657 21598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:03.968679 21598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:03.969877 21598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:03.969918 21598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:03.969944 21598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:54:03.971388 21598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.131.keytab
--principal=oryx/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:36527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=34151
--tserver_master_addrs=127.14.14.190:35807,127.14.14.189:39427,127.14.14.188:39071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:03.971598 21598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:03.971815 21598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:03.977139 21606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:03.977226 21604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:03.977138 21603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:03.977286 21598 server_base.cc:1034] running on GCE node
I20260928 22:54:03.977630 21598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:03.977905 21598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:03.979063 21598 hybrid_clock.cc:648] HybridClock initialized: now 1790636043979054 us; error 38 us; skew 500 ppm
Sep 28 22:54:03 dist-test-slave-kzfr krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:03.981443 21598 init.cc:377] Logged in from keytab as oryx/127.14.14.131@KRBTEST.COM (short username oryx)
I20260928 22:54:03.981974 21598 webserver.cc:460] Webserver started at http://127.14.14.131:34151/ using document root <none> and password file <none>
I20260928 22:54:03.982161 21598 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:03.982215 21598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:03.983467 21598 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:03.984078 21613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.984373 21598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:03.984467 21598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "5fb9e57173d94423a7e31e876116de53"
format_stamp: "Formatted at 2026-09-28 22:54:03 on dist-test-slave-kzfr"
server_key: "65a8670def992e2dd63b9b576f776bb5"
server_key_iv: "ce854afa34af983c9b426c3f7cd7c474"
server_key_version: "encryptionkey@0"
I20260928 22:54:03.984791 21598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:03.991905 21598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:03.993764 21598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:03.993906 21598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:03.994160 21598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:03.994513 21598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:03.994581 21598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.994614 21598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:03.994638 21598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:03.998085 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.994099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39071 (local address 127.0.0.1:44932)
0928 22:54:03.994153 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:03.994167 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:03.994218 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:03.994429 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:03.994430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:03.994432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:03.994485 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.994488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.995467 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.995468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.996500 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:03.996505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.997712 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.997714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:03.997822 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.997827 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:54:03.997898 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 22:54:03.997974 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:54:03.998164 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:03.994171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.14.14.188:39071)
0928 22:54:03.994222 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:03.994224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:03.994265 (+ 41us) server_negotiation.cc:407] Connection header received
0928 22:54:03.994296 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:03.994297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:03.994321 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:03.994395 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:03.994568 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.995362 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.996609 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:03.997364 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:03.997556 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:03.997675 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:03.997920 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 22:54:03.998020 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:54:03.998449 21278 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 22:54:03.998680 21314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca8e77469ae4f95a452cf2b744ddfdf. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } } }
I20260928 22:54:03.998837 21314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [sys.catalog]: This master's current role is: LEARNER
I20260928 22:54:04.000988 21598 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:36527
I20260928 22:54:04.001050 21726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:36527 every 8 connection(s)
I20260928 22:54:04.001474 21598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:54:04.002449 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 21598
Sep 28 22:54:04 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.188@KRBTEST.COM
I20260928 22:54:04.012904 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.003515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.14.14.188:39071)
0928 22:54:04.003704 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.003706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.004054 (+ 348us) server_negotiation.cc:407] Connection header received
0928 22:54:04.004151 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.004152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.004163 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.004202 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:04.005090 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.005918 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.006981 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.007275 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.007314 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.009821 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:04.009823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:04.009823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:04.009831 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:04.011593 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.012015 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.012016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.012016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.012078 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.012288 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.012288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.012289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.012413 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:04.012488 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.012677 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.012772 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
Sep 28 22:54:04 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.189@KRBTEST.COM
I20260928 22:54:04.014783 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.003875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43822 (local address 127.14.14.190:35807)
0928 22:54:04.004587 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.004589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.004596 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:54:04.004618 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.004619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.004632 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.004710 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:04.005489 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.006285 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.007381 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.007668 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.007957 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.009625 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:04.009626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:04.009627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:04.009634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:04.012452 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.013693 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.013694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.013695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.013762 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.013985 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.013986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.013986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.014120 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:04.014506 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.014546 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.014628 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":88,"threads_started":1}
I20260928 22:54:04.016234 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.002293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:55255 (local address 127.14.14.188:39071)
0928 22:54:04.002334 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.002336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.004319 (+ 1983us) server_negotiation.cc:407] Connection header received
0928 22:54:04.005658 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.005659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.005673 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.005701 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:04.006587 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.007285 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.008779 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.009093 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.009140 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.012441 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:04.012442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:04.012442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:04.012449 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:04.014304 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.015046 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.015047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.015048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.015109 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.015450 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.015451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.015451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.015601 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:04.015646 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.016050 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.016134 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:54:04.016566 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.003602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.14.14.189:39427)
0928 22:54:04.003863 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.003865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.003983 (+ 118us) server_negotiation.cc:407] Connection header received
0928 22:54:04.004096 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.004097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.004111 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.004151 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:04.005312 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.006052 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.007000 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.007318 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.007356 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.011490 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:04.011491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:04.011492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:04.011499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:04.014033 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.015382 (+ 1349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.015383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.015383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.015450 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.015726 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.015728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.015728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.015884 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:04.015961 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.016380 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.016461 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":44,"threads_started":1}
I20260928 22:54:04.016844 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.002221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39071 (local address 127.14.14.131:55255)
0928 22:54:04.004204 (+ 1983us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:04.004248 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:04.005529 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:04.005875 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:04.005879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:04.006204 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:04.006490 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.006496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.007395 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.007395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.008672 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.008676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.009232 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.009233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.009389 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.009899 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:04.009908 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:04.012328 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:04.014382 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:04.014389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:04.014391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:04.014695 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:04.015185 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:04.015186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:04.015186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:04.015365 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:04.015700 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:04.015707 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:04.015863 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 22:54:04.016365 (+ 502us) client_negotiation.cc:240] Negotiation successful
0928 22:54:04.016557 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1938,"mutex_wait_us":2,"thread_start_us":1794,"threads_started":1}
I20260928 22:54:04.017441 21727 heartbeater.cc:347] Connected to a master server at 127.14.14.188:39071
I20260928 22:54:04.017556 21727 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:04.017751 21727 heartbeater.cc:511] Master 127.14.14.188:39071 requested a full tablet report, sending...
I20260928 22:54:04.018203 21275 ts_manager.cc:194] Registered new tserver with Master: 5fb9e57173d94423a7e31e876116de53 (127.14.14.131:36527)
I20260928 22:54:04.018339 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.002222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:39427 (local address 127.14.14.131:35361)
0928 22:54:04.004447 (+ 2225us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:04.004459 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:04.007514 (+ 3055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:04.007737 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:04.007737 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:04.007954 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:04.008049 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.008051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.008983 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.008984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.009950 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.009952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.010507 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.010508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.010579 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.013530 (+ 2951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:04.013535 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:04.015113 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:04.017074 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:04.017075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:04.017075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:04.017465 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:04.017697 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:04.017698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:04.017699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:04.017756 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:04.018057 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:04.018057 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:04.018093 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 22:54:04.018149 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:54:04.018237 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2196,"mutex_wait_us":25,"thread_start_us":1866,"threads_started":1}
I20260928 22:54:04.018366 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.002440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:35361 (local address 127.14.14.189:39427)
0928 22:54:04.002495 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.002497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.004509 (+ 2012us) server_negotiation.cc:407] Connection header received
0928 22:54:04.007603 (+ 3094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.007604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.007615 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.007649 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:04.008141 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.008884 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.010047 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.010358 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.010404 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.015206 (+ 4802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:04.015207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:04.015208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:04.015214 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:04.016835 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.017547 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.017548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.017548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.017625 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.017838 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.017839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.017840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.017948 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:04.018011 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.018196 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.018278 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:54:04.018734 21729 heartbeater.cc:347] Connected to a master server at 127.14.14.189:39427
I20260928 22:54:04.018790 21729 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:04.018852 21729 heartbeater.cc:511] Master 127.14.14.189:39427 requested a full tablet report, sending...
I20260928 22:54:04.019095 21089 ts_manager.cc:194] Registered new tserver with Master: 5fb9e57173d94423a7e31e876116de53 (127.14.14.131:36527)
I20260928 22:54:04.017680 20490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:54:04.019915 20490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:54:04.020612 21050 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: NON_VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: true } } }
I20260928 22:54:04.020702 21050 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:04.021993 21051 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf: attempting to promote NON_VOTER 43f4101c5f22495b9950e8779f0fa717 to VOTER
I20260928 22:54:04.022107 21051 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } }
Sep 28 22:54:04 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636043, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
I20260928 22:54:04.022816 21098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 1 FOLLOWER]: Refusing update from remote peer bca8e77469ae4f95a452cf2b744ddfdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 22:54:04.022817 21278 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 1 LEARNER]: Refusing update from remote peer bca8e77469ae4f95a452cf2b744ddfdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 22:54:04.023016 21050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 22:54:04.023715 21050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 22:54:04.025098 21050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 43f4101c5f22495b9950e8779f0fa717 (127.14.14.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } } }
I20260928 22:54:04.025318 21099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 43f4101c5f22495b9950e8779f0fa717 (127.14.14.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } } }
I20260928 22:54:04.026082 21050 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } } }
I20260928 22:54:04.026175 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.002221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.131:54645)
0928 22:54:04.004225 (+ 2004us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:04.004248 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:04.009768 (+ 5520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:04.011602 (+ 1834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:04.011603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:04.011778 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:04.011855 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.011857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.013129 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.013130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.016244 (+ 3114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.016246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.020843 (+ 4597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.020844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.020911 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.021319 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:04.021324 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:04.023153 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:04.025047 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:04.025048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:04.025048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:04.025268 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:04.025469 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:04.025470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:04.025471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:04.025536 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:04.025811 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:04.025812 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:04.025855 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 22:54:04.025933 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:54:04.026056 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1935,"spinlock_wait_cycles":8576,"thread_start_us":1360,"threads_started":1}
I20260928 22:54:04.026180 21050 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:04.026221 21128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } } }
I20260928 22:54:04.026249 21050 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } } }
I20260928 22:54:04.026298 21128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:54:04.026306 21050 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:04.026343 21050 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } } }
I20260928 22:54:04.026391 21050 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:04.026185 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.002370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:54645 (local address 127.14.14.190:35807)
0928 22:54:04.002419 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.002421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.004320 (+ 1899us) server_negotiation.cc:407] Connection header received
0928 22:54:04.011423 (+ 7103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.011424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.011448 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.011492 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:04.011947 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.012675 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.020417 (+ 7742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.020695 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.020786 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.023236 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:04.023237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:04.023238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:04.023245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:04.024967 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.025328 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.025329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.025329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.025384 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.025629 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.025629 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.025630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.025742 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:04.025819 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.025972 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.026055 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:54:04.026573 21278 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 43f4101c5f22495b9950e8779f0fa717 (127.14.14.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } } }
I20260928 22:54:04.027097 21728 heartbeater.cc:347] Connected to a master server at 127.14.14.190:35807
I20260928 22:54:04.027143 21728 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:04.027202 21728 heartbeater.cc:511] Master 127.14.14.190:35807 requested a full tablet report, sending...
I20260928 22:54:04.027667 21314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } } }
I20260928 22:54:04.027755 21314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:54:04.027882 20490 ts_manager.cc:194] Re-registered known tserver with Master: 5fb9e57173d94423a7e31e876116de53 (127.14.14.131:36527)
I20260928 22:54:04.028317 20490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:54645
I20260928 22:54:04.031535 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.027263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:45887 (local address 127.0.0.1:38158)
0928 22:54:04.027561 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:04.027571 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:04.027612 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:04.027851 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:04.027852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:04.027854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:04.027896 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.027897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.029341 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.029342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.030420 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.030423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.031318 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.031319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.031381 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.031385 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:54:04.031414 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 22:54:04.031450 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":69,"threads_started":1}
I20260928 22:54:04.031958 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.027211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:42869 (local address 127.0.0.1:45780)
0928 22:54:04.027449 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:04.027470 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:04.027522 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:04.028242 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:04.028243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:04.028245 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:04.028291 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.028293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.029672 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.029673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.030751 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.030753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.031709 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.031709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.031779 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.031782 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:54:04.031841 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:54:04.031883 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260928 22:54:04.032084 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.027461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.14.14.129:45887)
0928 22:54:04.027650 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.027655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.027666 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:54:04.027709 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.027711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.027728 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.027849 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:04.028000 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.029242 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.030518 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.031220 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.031658 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.031746 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.031775 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.031822 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":72,"threads_started":1}
I20260928 22:54:04.031989 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.027701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.14.14.130:42869)
0928 22:54:04.028066 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.028071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.028082 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:54:04.028122 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.028124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.028141 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.028370 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:04.028431 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.029535 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.030836 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.031538 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.031707 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.031781 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.031852 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.031894 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":80,"threads_started":1}
I20260928 22:54:04.032732 21388 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3c00dad7654a8fa40a8797bf76d907 (DEFAULT_TABLE table=test-table [id=2b1fb5fefcc6421c953b62de97188122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:04.032799 21525 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3c00dad7654a8fa40a8797bf76d907 (DEFAULT_TABLE table=test-table [id=2b1fb5fefcc6421c953b62de97188122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:04.033073 21525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3c00dad7654a8fa40a8797bf76d907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:04.033092 21388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3c00dad7654a8fa40a8797bf76d907. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 22:54:04 dist-test-slave-kzfr krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636042, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.190@KRBTEST.COM for oryx/127.14.14.131@KRBTEST.COM
I20260928 22:54:04.036374 21756 tablet_bootstrap.cc:492] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e: Bootstrap starting.
I20260928 22:54:04.037120 21757 tablet_bootstrap.cc:492] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8: Bootstrap starting.
I20260928 22:54:04.037277 21756 tablet_bootstrap.cc:654] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:04.037750 21751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.027334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:36527 (local address 127.0.0.1:60970)
0928 22:54:04.028404 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:04.028415 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:04.028491 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:04.028704 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:04.028704 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:04.028877 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:04.028916 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.028918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.030020 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.030021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.030552 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.030554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.031295 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.031296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.031352 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.031882 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:04.031887 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:04.033986 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:04.036226 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:04.036227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:04.036228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:04.036452 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:04.036725 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:04.036726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:04.036726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:04.037000 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:04.037365 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:04.037366 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:04.037411 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 22:54:04.037507 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 22:54:04.037634 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"thread_start_us":67,"threads_started":1}
I20260928 22:54:04.037837 21757 tablet_bootstrap.cc:654] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:04.037814 21755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.027766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.14.14.131:36527)
0928 22:54:04.028325 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.028330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.028463 (+ 133us) server_negotiation.cc:407] Connection header received
0928 22:54:04.028570 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.028573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.028586 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.028627 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:04.029012 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.029943 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.030631 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.031207 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.031296 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.034104 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:04.034112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:04.034113 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:04.034120 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:04.036132 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.036560 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.036564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.036564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.036639 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.037099 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.037100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.037100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.037278 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:04.037367 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.037559 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.037673 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":80,"threads_started":1}
I20260928 22:54:04.038110 21757 log.cc:824] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:04.037736 21756 log.cc:824] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:04.038774 21659 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3c00dad7654a8fa40a8797bf76d907 (DEFAULT_TABLE table=test-table [id=2b1fb5fefcc6421c953b62de97188122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:04.038909 21757 tablet_bootstrap.cc:492] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8: No bootstrap required, opened a new log
I20260928 22:54:04.038966 21757 ts_tablet_manager.cc:1397] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:54:04.039112 21659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3c00dad7654a8fa40a8797bf76d907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:04.041558 21757 raft_consensus.cc:348] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } } peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } }
I20260928 22:54:04.041756 21757 raft_consensus.cc:374] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:04.041797 21757 raft_consensus.cc:729] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c89c8d94bf79445189e99922cdefdef8, State: Initialized, Role: FOLLOWER
I20260928 22:54:04.041906 21757 consensus_queue.cc:260] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } } peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } }
I20260928 22:54:04.042198 21756 tablet_bootstrap.cc:492] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e: No bootstrap required, opened a new log
I20260928 22:54:04.042207 21592 heartbeater.cc:503] Master 127.14.14.190:35807 was elected leader, sending a full tablet report...
I20260928 22:54:04.042263 21756 ts_tablet_manager.cc:1397] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 22:54:04.042820 21757 ts_tablet_manager.cc:1428] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260928 22:54:04.043393 21761 tablet_bootstrap.cc:492] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53: Bootstrap starting.
I20260928 22:54:04.044020 21761 tablet_bootstrap.cc:654] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:04.044320 21761 log.cc:824] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:04.044552 21756 raft_consensus.cc:348] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } } peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } }
I20260928 22:54:04.044687 21756 raft_consensus.cc:374] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:04.044708 21756 raft_consensus.cc:729] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae618bb3e59841e0a60be96d1b74235e, State: Initialized, Role: FOLLOWER
I20260928 22:54:04.044827 21756 consensus_queue.cc:260] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } } peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } }
I20260928 22:54:04.045053 21761 tablet_bootstrap.cc:492] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53: No bootstrap required, opened a new log
I20260928 22:54:04.045132 21455 heartbeater.cc:503] Master 127.14.14.190:35807 was elected leader, sending a full tablet report...
I20260928 22:54:04.045286 21761 ts_tablet_manager.cc:1397] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:54:04.045295 21756 ts_tablet_manager.cc:1428] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:54:04.047515 21761 raft_consensus.cc:348] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } } peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } }
I20260928 22:54:04.047631 21761 raft_consensus.cc:374] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:04.047652 21761 raft_consensus.cc:729] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb9e57173d94423a7e31e876116de53, State: Initialized, Role: FOLLOWER
I20260928 22:54:04.047780 21761 consensus_queue.cc:260] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } } peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } }
I20260928 22:54:04.048024 21761 ts_tablet_manager.cc:1428] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:54:04.048087 21728 heartbeater.cc:503] Master 127.14.14.190:35807 was elected leader, sending a full tablet report...
W20260928 22:54:04.122716 21594 tablet.cc:2369] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:54:04.160029 21760 raft_consensus.cc:482] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:54:04.160195 21760 raft_consensus.cc:504] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } } peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } }
I20260928 22:54:04.160607 21760 leader_election.cc:288] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae618bb3e59841e0a60be96d1b74235e (127.14.14.129:45887), 5fb9e57173d94423a7e31e876116de53 (127.14.14.131:36527)
I20260928 22:54:04.160934 21763 raft_consensus.cc:482] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:54:04.161049 21763 raft_consensus.cc:504] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } } peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } }
I20260928 22:54:04.161366 21763 leader_election.cc:288] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c89c8d94bf79445189e99922cdefdef8 (127.14.14.130:42869), 5fb9e57173d94423a7e31e876116de53 (127.14.14.131:36527)
I20260928 22:54:04.164520 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.160818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:45887 (local address 127.14.14.130:45193)
0928 22:54:04.160950 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:04.160961 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:04.161022 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:04.161191 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:04.161192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:04.161195 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:04.161270 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.161278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.162214 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.162214 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.163536 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.163538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.164258 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.164259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.164334 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.164337 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:54:04.164382 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 22:54:04.164426 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":62}
I20260928 22:54:04.164544 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.160800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:36527 (local address 127.14.14.130:37485)
0928 22:54:04.160915 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:04.160928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:04.160975 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:04.161255 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:04.161256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:04.161258 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:04.161322 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.161324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.162337 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.162337 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.163510 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.163512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.164306 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.164307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.164383 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.164387 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:54:04.164415 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 22:54:04.164450 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 22:54:04.164775 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.160982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:45193 (local address 127.14.14.129:45887)
0928 22:54:04.161024 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.161025 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.161034 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:54:04.161081 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.161081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.161093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.161140 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:04.161351 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.162131 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.163664 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.164161 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.164222 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.164229 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.164685 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.164719 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:54:04.164872 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.161602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:42869 (local address 127.14.14.129:58407)
0928 22:54:04.161670 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:04.161680 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:04.161731 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:04.161882 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:04.161883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:04.161884 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:04.161947 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.161948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.162792 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.162793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.163912 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.163915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.164618 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.164619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.164708 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.164714 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:54:04.164768 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 22:54:04.164815 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 22:54:04.165050 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.161514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:58407 (local address 127.14.14.130:42869)
0928 22:54:04.161548 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.161549 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.161782 (+ 233us) server_negotiation.cc:407] Connection header received
0928 22:54:04.161802 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.161803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.161811 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.161850 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:04.162011 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.162668 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.164010 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.164518 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.164578 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.164586 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.164938 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.164978 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:54:04.165056 21408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3c00dad7654a8fa40a8797bf76d907" candidate_uuid: "c89c8d94bf79445189e99922cdefdef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae618bb3e59841e0a60be96d1b74235e" is_pre_election: true
I20260928 22:54:04.165158 21408 raft_consensus.cc:2454] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c89c8d94bf79445189e99922cdefdef8 in term 0.
I20260928 22:54:04.165275 21545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3c00dad7654a8fa40a8797bf76d907" candidate_uuid: "ae618bb3e59841e0a60be96d1b74235e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89c8d94bf79445189e99922cdefdef8" is_pre_election: true
I20260928 22:54:04.165342 21478 leader_election.cc:302] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae618bb3e59841e0a60be96d1b74235e, c89c8d94bf79445189e99922cdefdef8; no voters:
I20260928 22:54:04.165381 21545 raft_consensus.cc:2454] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae618bb3e59841e0a60be96d1b74235e in term 0.
I20260928 22:54:04.165403 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.161717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:36527 (local address 127.14.14.129:44443)
0928 22:54:04.161754 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:04.161763 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:04.161794 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:04.162166 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:04.162167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:04.162169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:04.162226 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.162227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.163261 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.163262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.164361 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:04.164363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.165164 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.165165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:04.165250 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.165253 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:54:04.165284 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 22:54:04.165317 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 22:54:04.165552 21342 leader_election.cc:302] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae618bb3e59841e0a60be96d1b74235e, c89c8d94bf79445189e99922cdefdef8; no voters:
I20260928 22:54:04.165688 21763 raft_consensus.cc:2781] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:54:04.165705 21760 raft_consensus.cc:2781] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:54:04.165755 21763 raft_consensus.cc:482] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:54:04.165760 21760 raft_consensus.cc:482] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:54:04.165779 21763 raft_consensus.cc:3037] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:04.165783 21760 raft_consensus.cc:3037] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:04.165879 21755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.160910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:37485 (local address 127.14.14.131:36527)
0928 22:54:04.161072 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.161073 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.161081 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:54:04.161105 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.161106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.161131 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.161221 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:04.161421 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.162177 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.163628 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.164221 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.165329 (+ 1108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.165686 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.165708 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.165749 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260928 22:54:04.165886 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.161888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:44443 (local address 127.14.14.131:36527)
0928 22:54:04.162042 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.162043 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.162050 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:54:04.162069 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.162069 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.162077 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.162318 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:04.162367 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.163165 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.164473 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.165064 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.165672 (+ 608us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.165708 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.165729 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.165770 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260928 22:54:04.166152 21681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3c00dad7654a8fa40a8797bf76d907" candidate_uuid: "c89c8d94bf79445189e99922cdefdef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fb9e57173d94423a7e31e876116de53" is_pre_election: true
I20260928 22:54:04.166157 21680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3c00dad7654a8fa40a8797bf76d907" candidate_uuid: "ae618bb3e59841e0a60be96d1b74235e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fb9e57173d94423a7e31e876116de53" is_pre_election: true
I20260928 22:54:04.166280 21681 raft_consensus.cc:2454] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c89c8d94bf79445189e99922cdefdef8 in term 0.
I20260928 22:54:04.166741 21763 raft_consensus.cc:504] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } } peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } }
I20260928 22:54:04.166752 21760 raft_consensus.cc:504] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } } peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } }
I20260928 22:54:04.166867 21763 leader_election.cc:288] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [CANDIDATE]: Term 1 election: Requested vote from peers c89c8d94bf79445189e99922cdefdef8 (127.14.14.130:42869), 5fb9e57173d94423a7e31e876116de53 (127.14.14.131:36527)
I20260928 22:54:04.166882 21760 leader_election.cc:288] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [CANDIDATE]: Term 1 election: Requested vote from peers ae618bb3e59841e0a60be96d1b74235e (127.14.14.129:45887), 5fb9e57173d94423a7e31e876116de53 (127.14.14.131:36527)
I20260928 22:54:04.167011 21681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3c00dad7654a8fa40a8797bf76d907" candidate_uuid: "ae618bb3e59841e0a60be96d1b74235e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fb9e57173d94423a7e31e876116de53"
I20260928 22:54:04.167078 21545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3c00dad7654a8fa40a8797bf76d907" candidate_uuid: "ae618bb3e59841e0a60be96d1b74235e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89c8d94bf79445189e99922cdefdef8"
I20260928 22:54:04.167081 21408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3c00dad7654a8fa40a8797bf76d907" candidate_uuid: "c89c8d94bf79445189e99922cdefdef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae618bb3e59841e0a60be96d1b74235e"
I20260928 22:54:04.167085 21681 raft_consensus.cc:3037] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:04.167093 21680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3c00dad7654a8fa40a8797bf76d907" candidate_uuid: "c89c8d94bf79445189e99922cdefdef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fb9e57173d94423a7e31e876116de53"
I20260928 22:54:04.167153 21408 raft_consensus.cc:2379] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c89c8d94bf79445189e99922cdefdef8 in current term 1: Already voted for candidate ae618bb3e59841e0a60be96d1b74235e in this term.
I20260928 22:54:04.167155 21545 raft_consensus.cc:2379] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ae618bb3e59841e0a60be96d1b74235e in current term 1: Already voted for candidate c89c8d94bf79445189e99922cdefdef8 in this term.
I20260928 22:54:04.167359 21478 leader_election.cc:302] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c89c8d94bf79445189e99922cdefdef8; no voters: 5fb9e57173d94423a7e31e876116de53, ae618bb3e59841e0a60be96d1b74235e
I20260928 22:54:04.167440 21760 raft_consensus.cc:2726] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 22:54:04.167918 21681 raft_consensus.cc:2454] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae618bb3e59841e0a60be96d1b74235e in term 1.
I20260928 22:54:04.168097 21342 leader_election.cc:302] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5fb9e57173d94423a7e31e876116de53, ae618bb3e59841e0a60be96d1b74235e; no voters: c89c8d94bf79445189e99922cdefdef8
I20260928 22:54:04.168192 21763 raft_consensus.cc:2781] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:54:04.168344 21763 raft_consensus.cc:686] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [term 1 LEADER]: Becoming Leader. State: Replica: ae618bb3e59841e0a60be96d1b74235e, State: Running, Role: LEADER
I20260928 22:54:04.168444 21763 consensus_queue.cc:237] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } } peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } }
I20260928 22:54:04.169224 20490 catalog_manager.cc:5495] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e reported cstate change: term changed from 0 to 1, leader changed from <none> to ae618bb3e59841e0a60be96d1b74235e (127.14.14.129). New cstate: current_term: 1 leader_uuid: "ae618bb3e59841e0a60be96d1b74235e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } health_report { overall_health: UNKNOWN } } }
I20260928 22:54:04.209739 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.207532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.14.14.129:45887)
0928 22:54:04.207590 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.207596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.207604 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:54:04.207640 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.207641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.207658 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.207721 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:54:04.207844 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.208305 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.209055 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.209245 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.209293 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.209421 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:54:04.209526 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:54:04.209640 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.209665 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.209683 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:54:04.211567 21545 raft_consensus.cc:1261] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [term 1 FOLLOWER]: Refusing update from remote peer ae618bb3e59841e0a60be96d1b74235e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:04.211570 21681 raft_consensus.cc:1261] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53 [term 1 FOLLOWER]: Refusing update from remote peer ae618bb3e59841e0a60be96d1b74235e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:04.211833 21766 consensus_queue.cc:1035] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:04.211995 21763 consensus_queue.cc:1035] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:04.214814 21771 mvcc.cc:205] Tried to move back new op lower bound from 7334445237087756288 to 7334445236914462720. Current Snapshot: MvccSnapshot[applied={T|T < 7334445237087756288}]
I20260928 22:54:04.215374 21769 mvcc.cc:205] Tried to move back new op lower bound from 7334445237087756288 to 7334445236914462720. Current Snapshot: MvccSnapshot[applied={T|T < 7334445237087756288}]
I20260928 22:54:04.215479 21770 mvcc.cc:205] Tried to move back new op lower bound from 7334445237087756288 to 7334445236914462720. Current Snapshot: MvccSnapshot[applied={T|T < 7334445237087756288}]
I20260928 22:54:04.218497 20491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:54:04.218592 20491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:54:04.220181 20491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bca8e77469ae4f95a452cf2b744ddfdf: Sending DeleteTablet for 3 replicas of tablet 8d3c00dad7654a8fa40a8797bf76d907
I20260928 22:54:04.220515 21659 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3c00dad7654a8fa40a8797bf76d907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:04 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:60970
I20260928 22:54:04.220520 21388 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3c00dad7654a8fa40a8797bf76d907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:04 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:38158
I20260928 22:54:04.220525 21525 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3c00dad7654a8fa40a8797bf76d907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:04 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:45780
I20260928 22:54:04.220796 21778 tablet_replica.cc:330] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e: stopping tablet replica
I20260928 22:54:04.220891 21779 tablet_replica.cc:330] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8: stopping tablet replica
I20260928 22:54:04.220898 21778 raft_consensus.cc:2229] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:54:04.221019 21778 raft_consensus.cc:2258] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:04.221025 21779 raft_consensus.cc:2229] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:54:04.221176 21779 raft_consensus.cc:2258] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:04.221478 21780 tablet_replica.cc:330] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53: stopping tablet replica
I20260928 22:54:04.221604 21780 raft_consensus.cc:2229] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:54:04.221704 21780 raft_consensus.cc:2258] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:04.221974 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 20459
I20260928 22:54:04.222015 21779 ts_tablet_manager.cc:1905] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:04.222892 21778 ts_tablet_manager.cc:1905] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:04.224319 21779 ts_tablet_manager.cc:1918] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:54:04.224399 21779 log.cc:1196] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/8d3c00dad7654a8fa40a8797bf76d907
I20260928 22:54:04.224689 21779 ts_tablet_manager.cc:1939] T 8d3c00dad7654a8fa40a8797bf76d907 P c89c8d94bf79445189e99922cdefdef8: Deleting consensus metadata
I20260928 22:54:04.224908 21778 ts_tablet_manager.cc:1918] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:54:04.225042 21778 log.cc:1196] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-0/wal/wals/8d3c00dad7654a8fa40a8797bf76d907
I20260928 22:54:04.225394 21778 ts_tablet_manager.cc:1939] T 8d3c00dad7654a8fa40a8797bf76d907 P ae618bb3e59841e0a60be96d1b74235e: Deleting consensus metadata
I20260928 22:54:04.225422 21780 ts_tablet_manager.cc:1905] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:04.226856 21780 ts_tablet_manager.cc:1918] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:54:04.226944 21780 log.cc:1196] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/8d3c00dad7654a8fa40a8797bf76d907
I20260928 22:54:04.227206 21780 ts_tablet_manager.cc:1939] T 8d3c00dad7654a8fa40a8797bf76d907 P 5fb9e57173d94423a7e31e876116de53: Deleting consensus metadata
W20260928 22:54:04.228035 21616 connection.cc:535] server connection from 127.0.0.1:60970 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 22:54:04.228052 21343 connection.cc:535] server connection from 127.0.0.1:38158 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 22:54:04.228062 21480 connection.cc:535] server connection from 127.0.0.1:45780 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 22:54:04.236945 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.229897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.14.14.188:39071)
0928 22:54:04.230608 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.230610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.230616 (+ 6us) server_negotiation.cc:407] Connection header received
0928 22:54:04.230634 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.230635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.230647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.230706 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:04.231243 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.231719 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.232422 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.232640 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.232714 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.234071 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:04.234072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:04.234072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:04.234078 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:04.235651 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.236066 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.236067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.236068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.236122 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.236314 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.236315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.236315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.236434 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:04.236537 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.236750 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.236832 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686}
I20260928 22:54:04.237138 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.221556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.14.14.188:39071)
0928 22:54:04.221634 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.221636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.221643 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:54:04.221664 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.221665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.221676 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.221730 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:04.229718 (+ 7988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.230519 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.231750 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.232062 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.232106 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.233771 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:04.233773 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:04.233773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:04.233781 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:04.235862 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.236252 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.236253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.236253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.236308 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.236536 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.236537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.236537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.236654 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:04.236724 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.236943 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.237036 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 22:54:04.241775 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:04.229733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.14.14.189:39427)
0928 22:54:04.229779 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:04.229781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:04.233767 (+ 3986us) server_negotiation.cc:407] Connection header received
0928 22:54:04.233870 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:04.233871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:04.233883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:04.233916 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:04.235513 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.236247 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.237735 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:04.237941 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:04.238027 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:04.239201 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:04.239202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:04.239203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:04.239210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:04.240600 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.240999 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.241000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.241000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.241057 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:04.241266 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:04.241267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:04.241267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:04.241399 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:04.241495 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:04.241596 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 22:54:04.241673 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:54:05.216174 21788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.215859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.131:55149)
0928 22:54:05.215999 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.216073 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
W20260928 22:54:05.216674 21728 heartbeater.cc:644] Failed to heartbeat to 127.14.14.190:35807 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
I20260928 22:54:05.216719 21789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.216421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.130:37471)
0928 22:54:05.216607 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.216675 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":65,"threads_started":1}
W20260928 22:54:05.216897 21592 heartbeater.cc:644] Failed to heartbeat to 127.14.14.190:35807 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
I20260928 22:54:05.217754 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.217445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.129:44101)
0928 22:54:05.217622 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.217687 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
W20260928 22:54:05.218037 21455 heartbeater.cc:644] Failed to heartbeat to 127.14.14.190:35807 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
I20260928 22:54:05.720945 21791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bca8e77469ae4f95a452cf2b744ddfdf)
I20260928 22:54:05.721114 21791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } }
I20260928 22:54:05.721417 21791 leader_election.cc:288] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bca8e77469ae4f95a452cf2b744ddfdf (127.14.14.190:35807), 75f5bd5a76c94b4bbf6b5e099dfa46ee (127.14.14.189:39427)
I20260928 22:54:05.721421 21792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bca8e77469ae4f95a452cf2b744ddfdf)
I20260928 22:54:05.721566 21792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } }
I20260928 22:54:05.721849 21792 leader_election.cc:288] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bca8e77469ae4f95a452cf2b744ddfdf (127.14.14.190:35807), 43f4101c5f22495b9950e8779f0fa717 (127.14.14.188:39071)
I20260928 22:54:05.721861 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.721506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:44342)
0928 22:54:05.721771 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.721820 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":51,"threads_started":1}
W20260928 22:54:05.722121 21262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
I20260928 22:54:05.722484 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.722121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:44356)
0928 22:54:05.722398 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.722450 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":42,"threads_started":1}
W20260928 22:54:05.722589 21076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
I20260928 22:54:05.722664 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.722441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:44360)
0928 22:54:05.722472 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.722501 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 22:54:05.722801 21262 leader_election.cc:334] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bca8e77469ae4f95a452cf2b744ddfdf (127.14.14.190:35807): Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
I20260928 22:54:05.723176 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.723097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:44368)
0928 22:54:05.723131 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.723151 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 22:54:05.723315 21076 leader_election.cc:334] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bca8e77469ae4f95a452cf2b744ddfdf (127.14.14.190:35807): Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
I20260928 22:54:05.725312 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.721619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:39427 (local address 127.0.0.1:52228)
0928 22:54:05.721954 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.721965 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:05.722018 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:05.722242 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:05.722243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:05.722246 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:05.722329 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:05.722331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.723279 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.723279 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:05.724392 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:05.724395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.725097 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.725098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:05.725159 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:05.725163 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:54:05.725191 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 22:54:05.725234 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":1,"thread_start_us":48,"threads_started":1}
I20260928 22:54:05.725647 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.721659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.14.14.189:39427)
0928 22:54:05.722108 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:05.722110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:05.722117 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:54:05.722140 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:05.722141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:05.722153 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:05.722397 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:05.722433 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.723185 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.724490 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.725015 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.725474 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:05.725488 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:05.725506 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 22:54:05.725546 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":61,"threads_started":1}
I20260928 22:54:05.725828 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43f4101c5f22495b9950e8779f0fa717" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" is_pre_election: true
I20260928 22:54:05.725960 21099 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f4101c5f22495b9950e8779f0fa717 in term 1.
I20260928 22:54:05.726048 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.721984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39071 (local address 127.0.0.1:44970)
0928 22:54:05.722272 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.722281 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:05.722331 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:05.722639 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:05.722640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:05.722642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:05.722695 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:05.722697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.723811 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.723812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:05.724957 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:05.724959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.725772 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.725773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:05.725856 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:05.725860 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:54:05.725913 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:54:05.725978 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":51,"threads_started":1}
I20260928 22:54:05.726054 21797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.722102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.14.14.188:39071)
0928 22:54:05.722180 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:05.722182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:05.722477 (+ 295us) server_negotiation.cc:407] Connection header received
0928 22:54:05.722512 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:05.722512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:05.722522 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:05.722593 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:05.722888 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.723705 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.725047 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.725666 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.725729 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:05.725741 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:05.725942 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 22:54:05.725992 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"thread_start_us":37,"threads_started":1}
I20260928 22:54:05.726157 21260 leader_election.cc:302] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43f4101c5f22495b9950e8779f0fa717, 75f5bd5a76c94b4bbf6b5e099dfa46ee; no voters: bca8e77469ae4f95a452cf2b744ddfdf
I20260928 22:54:05.726275 21791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 22:54:05.726315 21791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bca8e77469ae4f95a452cf2b744ddfdf)
I20260928 22:54:05.726336 21791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:54:05.726418 21278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "43f4101c5f22495b9950e8779f0fa717" is_pre_election: true
I20260928 22:54:05.727156 21791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } }
I20260928 22:54:05.727313 21791 leader_election.cc:288] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [CANDIDATE]: Term 2 election: Requested vote from peers bca8e77469ae4f95a452cf2b744ddfdf (127.14.14.190:35807), 75f5bd5a76c94b4bbf6b5e099dfa46ee (127.14.14.189:39427)
I20260928 22:54:05.727322 21278 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75f5bd5a76c94b4bbf6b5e099dfa46ee in current term 2: Already voted for candidate 43f4101c5f22495b9950e8779f0fa717 in this term.
I20260928 22:54:05.727456 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43f4101c5f22495b9950e8779f0fa717" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee"
I20260928 22:54:05.727532 21099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:54:05.727541 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.727400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:44384)
0928 22:54:05.727489 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.727515 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 22:54:05.727545 21075 leader_election.cc:302] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 75f5bd5a76c94b4bbf6b5e099dfa46ee; no voters: 43f4101c5f22495b9950e8779f0fa717, bca8e77469ae4f95a452cf2b744ddfdf
I20260928 22:54:05.727955 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.727853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:44394)
0928 22:54:05.727905 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.727933 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 22:54:05.728032 21262 leader_election.cc:334] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bca8e77469ae4f95a452cf2b744ddfdf (127.14.14.190:35807): Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
I20260928 22:54:05.728325 21099 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f4101c5f22495b9950e8779f0fa717 in term 2.
I20260928 22:54:05.728406 21792 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:54:05.728626 21260 leader_election.cc:302] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43f4101c5f22495b9950e8779f0fa717, 75f5bd5a76c94b4bbf6b5e099dfa46ee; no voters: bca8e77469ae4f95a452cf2b744ddfdf
I20260928 22:54:05.728771 21791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:54:05.728929 21791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [term 2 LEADER]: Becoming Leader. State: Replica: 43f4101c5f22495b9950e8779f0fa717, State: Running, Role: LEADER
I20260928 22:54:05.729022 21791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } }
I20260928 22:54:05.729358 21801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43f4101c5f22495b9950e8779f0fa717. Latest consensus state: current_term: 2 leader_uuid: "43f4101c5f22495b9950e8779f0fa717" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } } }
I20260928 22:54:05.729461 21801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:05.729682 21803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:54:05.730082 21803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:54:05.730175 21803 catalog_manager.cc:1259] Loaded cluster ID: f7c612814c4549cf84628db94f6e15d2
I20260928 22:54:05.730207 21803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:54:05.730367 21803 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:54:05.730463 21803 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: Loaded TSK: 0
I20260928 22:54:05.730607 21803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:54:05.778656 21275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:54:05.780246 21275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:54:05.781430 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.781238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:44408)
0928 22:54:05.781354 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.781395 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 22:54:05.781418 21099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [term 2 FOLLOWER]: Refusing update from remote peer 43f4101c5f22495b9950e8779f0fa717: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 22:54:05.781711 21801 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 22:54:05.781838 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.781725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:44412)
0928 22:54:05.781779 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.781812 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 22:54:05.781939 21262 consensus_peers.cc:487] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 -> Peer bca8e77469ae4f95a452cf2b744ddfdf (127.14.14.190:35807): Couldn't send request to peer bca8e77469ae4f95a452cf2b744ddfdf. Status: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 22:54:05.782457 21792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43f4101c5f22495b9950e8779f0fa717. Latest consensus state: current_term: 2 leader_uuid: "43f4101c5f22495b9950e8779f0fa717" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } } }
I20260928 22:54:05.782579 21792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:54:05.782998 21791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "43f4101c5f22495b9950e8779f0fa717" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } } }
I20260928 22:54:05.783092 21791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717 [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:05.783188 21792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "43f4101c5f22495b9950e8779f0fa717" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bca8e77469ae4f95a452cf2b744ddfdf" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35807 } } peers { permanent_uuid: "75f5bd5a76c94b4bbf6b5e099dfa46ee" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 39427 } attrs { promote: false } } peers { permanent_uuid: "43f4101c5f22495b9950e8779f0fa717" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39071 } attrs { promote: false } } }
I20260928 22:54:05.783272 21792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f5bd5a76c94b4bbf6b5e099dfa46ee [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:54:05.788254 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.784903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:36527 (local address 127.0.0.1:60982)
0928 22:54:05.784953 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.784964 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:05.785020 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:05.785366 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:05.785367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:05.785369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:05.785418 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:05.785419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.786203 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.786204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:05.787282 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:05.787284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.788022 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.788023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:05.788083 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:05.788087 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:54:05.788133 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 22:54:05.788179 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:54:05.788277 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.784973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60982 (local address 127.14.14.131:36527)
0928 22:54:05.785233 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:05.785235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:05.785244 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:54:05.785271 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:05.785273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:05.785284 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:05.785356 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:05.785484 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.786114 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.787378 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.787929 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.788023 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:05.788035 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:05.788152 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 22:54:05.788202 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":56,"threads_started":1}
I20260928 22:54:05.788312 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.784975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:45887 (local address 127.0.0.1:38172)
0928 22:54:05.785067 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.785088 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:05.785141 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:05.785409 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:05.785410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:05.785412 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:05.785443 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:05.785444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.786415 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.786416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:05.787247 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:05.787249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.788074 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.788075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:05.788134 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:05.788137 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:54:05.788183 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 22:54:05.788212 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 22:54:05.788587 21659 tablet_service.cc:1467] Processing CreateTablet for tablet d47f7cf6902a47708761e5da7a8231d6 (DEFAULT_TABLE table=test-table [id=8f25b56cb833442e84ce6aa6727d4f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:05.788728 21659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d47f7cf6902a47708761e5da7a8231d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:05.788807 21812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.785075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.14.14.129:45887)
0928 22:54:05.785288 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:05.785291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:05.785298 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:54:05.785336 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:05.785336 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:05.785348 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:05.785408 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:05.785548 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.786323 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.787361 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.787976 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.788072 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:05.788082 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:05.788613 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 22:54:05.788703 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":70,"threads_started":1}
I20260928 22:54:05.788853 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.785008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:42869 (local address 127.0.0.1:45796)
0928 22:54:05.785121 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.785131 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:05.785173 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:05.785627 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:05.785628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:05.785630 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:05.785670 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:05.785672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.786534 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.786535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:05.787714 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:05.787716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.788570 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.788572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:05.788642 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:05.788646 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:54:05.788721 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 22:54:05.788760 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":78,"threads_started":1}
I20260928 22:54:05.789053 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.785192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45796 (local address 127.14.14.130:42869)
0928 22:54:05.785444 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:05.785446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:05.785464 (+ 18us) server_negotiation.cc:407] Connection header received
0928 22:54:05.785488 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:05.785489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:05.785499 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:05.785556 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:05.785754 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.786451 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.787795 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:05.788410 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:05.788534 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:05.788546 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:05.788936 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 22:54:05.788977 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":55,"threads_started":1}
I20260928 22:54:05.789110 21388 tablet_service.cc:1467] Processing CreateTablet for tablet d47f7cf6902a47708761e5da7a8231d6 (DEFAULT_TABLE table=test-table [id=8f25b56cb833442e84ce6aa6727d4f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:05.789207 21388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d47f7cf6902a47708761e5da7a8231d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:05.789845 21525 tablet_service.cc:1467] Processing CreateTablet for tablet d47f7cf6902a47708761e5da7a8231d6 (DEFAULT_TABLE table=test-table [id=8f25b56cb833442e84ce6aa6727d4f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:05.789952 21525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d47f7cf6902a47708761e5da7a8231d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:05.790341 21814 tablet_bootstrap.cc:492] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53: Bootstrap starting.
I20260928 22:54:05.790681 21815 tablet_bootstrap.cc:492] T d47f7cf6902a47708761e5da7a8231d6 P ae618bb3e59841e0a60be96d1b74235e: Bootstrap starting.
I20260928 22:54:05.790740 21814 tablet_bootstrap.cc:654] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:05.791078 21815 tablet_bootstrap.cc:654] T d47f7cf6902a47708761e5da7a8231d6 P ae618bb3e59841e0a60be96d1b74235e: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:05.791297 21817 tablet_bootstrap.cc:492] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8: Bootstrap starting.
I20260928 22:54:05.791359 21814 tablet_bootstrap.cc:492] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53: No bootstrap required, opened a new log
I20260928 22:54:05.791415 21814 ts_tablet_manager.cc:1397] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:05.791553 21814 raft_consensus.cc:348] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } } peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } }
I20260928 22:54:05.791646 21814 raft_consensus.cc:374] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:05.791671 21814 raft_consensus.cc:729] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb9e57173d94423a7e31e876116de53, State: Initialized, Role: FOLLOWER
I20260928 22:54:05.791724 21814 consensus_queue.cc:260] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } } peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } }
I20260928 22:54:05.791749 21817 tablet_bootstrap.cc:654] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:05.791882 21814 ts_tablet_manager.cc:1428] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:05.792289 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.792078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.131:48645)
0928 22:54:05.792215 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.792252 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":98,"threads_started":1}
I20260928 22:54:05.792294 21815 tablet_bootstrap.cc:492] T d47f7cf6902a47708761e5da7a8231d6 P ae618bb3e59841e0a60be96d1b74235e: No bootstrap required, opened a new log
I20260928 22:54:05.792410 21815 ts_tablet_manager.cc:1397] T d47f7cf6902a47708761e5da7a8231d6 P ae618bb3e59841e0a60be96d1b74235e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:54:05.792459 21817 tablet_bootstrap.cc:492] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8: No bootstrap required, opened a new log
I20260928 22:54:05.792514 21817 ts_tablet_manager.cc:1397] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:05.792634 21817 raft_consensus.cc:348] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } } peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } }
I20260928 22:54:05.792708 21817 raft_consensus.cc:374] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:05.792743 21817 raft_consensus.cc:729] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c89c8d94bf79445189e99922cdefdef8, State: Initialized, Role: FOLLOWER
I20260928 22:54:05.792790 21817 consensus_queue.cc:260] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } } peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } }
I20260928 22:54:05.792841 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.792755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.131:46463)
0928 22:54:05.792788 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.792815 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 22:54:05.792811 21815 raft_consensus.cc:348] T d47f7cf6902a47708761e5da7a8231d6 P ae618bb3e59841e0a60be96d1b74235e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } } peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } }
I20260928 22:54:05.792943 21815 raft_consensus.cc:374] T d47f7cf6902a47708761e5da7a8231d6 P ae618bb3e59841e0a60be96d1b74235e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:05.792949 21817 ts_tablet_manager.cc:1428] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:05.792950 21727 heartbeater.cc:503] Master 127.14.14.188:39071 was elected leader, sending a full tablet report...
I20260928 22:54:05.792996 21815 raft_consensus.cc:729] T d47f7cf6902a47708761e5da7a8231d6 P ae618bb3e59841e0a60be96d1b74235e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae618bb3e59841e0a60be96d1b74235e, State: Initialized, Role: FOLLOWER
I20260928 22:54:05.793077 21815 consensus_queue.cc:260] T d47f7cf6902a47708761e5da7a8231d6 P ae618bb3e59841e0a60be96d1b74235e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } } peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } }
I20260928 22:54:05.793295 21815 ts_tablet_manager.cc:1428] T d47f7cf6902a47708761e5da7a8231d6 P ae618bb3e59841e0a60be96d1b74235e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 22:54:05.793431 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.793220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.130:36229)
0928 22:54:05.793345 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.793382 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
I20260928 22:54:05.793653 21591 heartbeater.cc:503] Master 127.14.14.188:39071 was elected leader, sending a full tablet report...
I20260928 22:54:05.793769 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.793689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.130:59833)
0928 22:54:05.793727 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.793751 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 22:54:05.794118 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:05.793862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.129:45439)
0928 22:54:05.794049 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:05.794084 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":118,"threads_started":1}
I20260928 22:54:05.794252 21454 heartbeater.cc:503] Master 127.14.14.188:39071 was elected leader, sending a full tablet report...
W20260928 22:54:05.988058 21457 tablet.cc:2369] T d47f7cf6902a47708761e5da7a8231d6 P ae618bb3e59841e0a60be96d1b74235e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:54:06.011184 21730 tablet.cc:2369] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:54:06.065223 21819 raft_consensus.cc:482] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:54:06.065323 21819 raft_consensus.cc:504] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } } peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } }
I20260928 22:54:06.065665 21819 leader_election.cc:288] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c89c8d94bf79445189e99922cdefdef8 (127.14.14.130:42869), ae618bb3e59841e0a60be96d1b74235e (127.14.14.129:45887)
I20260928 22:54:06.068928 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:06.066012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:46399 (local address 127.14.14.130:42869)
0928 22:54:06.066060 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:06.066062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:06.066167 (+ 105us) server_negotiation.cc:407] Connection header received
0928 22:54:06.066254 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:06.066256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:06.066265 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:06.066338 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:06.066483 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:06.066975 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:06.068197 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:06.068567 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:06.068653 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:06.068664 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:06.068815 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 22:54:06.068852 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:54:06.068960 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:06.065872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:42869 (local address 127.14.14.131:46399)
0928 22:54:06.066084 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:06.066095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:06.066186 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:06.066342 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:06.066343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:06.066345 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:06.066403 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:06.066405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:06.067066 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:06.067068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:06.068115 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:06.068118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:06.068653 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:06.068654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:06.068717 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:06.068721 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:54:06.068804 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:54:06.068844 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":28}
I20260928 22:54:06.068954 21812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:06.065929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:35799 (local address 127.14.14.129:45887)
0928 22:54:06.066009 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:06.066012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:06.066243 (+ 231us) server_negotiation.cc:407] Connection header received
0928 22:54:06.066305 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:06.066306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:06.066316 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:06.066373 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:06.066533 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:06.067005 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:06.068225 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:06.068596 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:06.068676 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:06.068686 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:06.068848 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 22:54:06.068885 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 22:54:06.068977 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:06.065872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:45887 (local address 127.14.14.131:35799)
0928 22:54:06.066131 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:06.066142 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:06.066246 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:06.066387 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:06.066387 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:06.066389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:06.066452 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:06.066453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:06.067091 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:06.067092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:06.068143 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:06.068146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:06.068685 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:06.068686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:06.068764 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:06.068775 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 22:54:06.068842 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 22:54:06.068881 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":35,"thread_start_us":138,"threads_started":1}
I20260928 22:54:06.069193 21545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47f7cf6902a47708761e5da7a8231d6" candidate_uuid: "5fb9e57173d94423a7e31e876116de53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89c8d94bf79445189e99922cdefdef8" is_pre_election: true
I20260928 22:54:06.069320 21545 raft_consensus.cc:2454] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fb9e57173d94423a7e31e876116de53 in term 0.
I20260928 22:54:06.069401 21408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47f7cf6902a47708761e5da7a8231d6" candidate_uuid: "5fb9e57173d94423a7e31e876116de53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae618bb3e59841e0a60be96d1b74235e" is_pre_election: true
I20260928 22:54:06.069530 21408 raft_consensus.cc:2454] T d47f7cf6902a47708761e5da7a8231d6 P ae618bb3e59841e0a60be96d1b74235e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fb9e57173d94423a7e31e876116de53 in term 0.
I20260928 22:54:06.069502 21615 leader_election.cc:302] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fb9e57173d94423a7e31e876116de53, c89c8d94bf79445189e99922cdefdef8; no voters:
I20260928 22:54:06.069684 21819 raft_consensus.cc:2781] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:54:06.069746 21819 raft_consensus.cc:482] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:54:06.069780 21819 raft_consensus.cc:3037] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:06.070465 21819 raft_consensus.cc:504] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } } peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } }
I20260928 22:54:06.070628 21819 leader_election.cc:288] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [CANDIDATE]: Term 1 election: Requested vote from peers c89c8d94bf79445189e99922cdefdef8 (127.14.14.130:42869), ae618bb3e59841e0a60be96d1b74235e (127.14.14.129:45887)
I20260928 22:54:06.070807 21408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47f7cf6902a47708761e5da7a8231d6" candidate_uuid: "5fb9e57173d94423a7e31e876116de53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae618bb3e59841e0a60be96d1b74235e"
I20260928 22:54:06.070807 21545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d47f7cf6902a47708761e5da7a8231d6" candidate_uuid: "5fb9e57173d94423a7e31e876116de53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89c8d94bf79445189e99922cdefdef8"
I20260928 22:54:06.070885 21408 raft_consensus.cc:3037] T d47f7cf6902a47708761e5da7a8231d6 P ae618bb3e59841e0a60be96d1b74235e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:06.070885 21545 raft_consensus.cc:3037] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:06.071653 21545 raft_consensus.cc:2454] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fb9e57173d94423a7e31e876116de53 in term 1.
I20260928 22:54:06.071684 21408 raft_consensus.cc:2454] T d47f7cf6902a47708761e5da7a8231d6 P ae618bb3e59841e0a60be96d1b74235e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fb9e57173d94423a7e31e876116de53 in term 1.
I20260928 22:54:06.071861 21614 leader_election.cc:302] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fb9e57173d94423a7e31e876116de53, ae618bb3e59841e0a60be96d1b74235e; no voters:
I20260928 22:54:06.071959 21819 raft_consensus.cc:2781] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:54:06.072129 21819 raft_consensus.cc:686] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [term 1 LEADER]: Becoming Leader. State: Replica: 5fb9e57173d94423a7e31e876116de53, State: Running, Role: LEADER
I20260928 22:54:06.072255 21819 consensus_queue.cc:237] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } } peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } }
I20260928 22:54:06.072449 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:06.072333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.131:35753)
0928 22:54:06.072383 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:06.072419 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 22:54:06.073108 21274 catalog_manager.cc:5495] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fb9e57173d94423a7e31e876116de53 (127.14.14.131). New cstate: current_term: 1 leader_uuid: "5fb9e57173d94423a7e31e876116de53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb9e57173d94423a7e31e876116de53" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 } health_report { overall_health: UNKNOWN } } }
I20260928 22:54:06.142283 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:06.139816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.14.14.131:36527)
0928 22:54:06.139860 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:06.139862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:06.139870 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:54:06.139933 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:06.139934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:06.139954 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:06.140026 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:54:06.140127 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:06.140594 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:06.141378 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:06.141569 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:06.141643 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:06.141774 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:54:06.141876 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:54:06.141930 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:06.142148 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 22:54:06.142179 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:54:06.143435 21545 raft_consensus.cc:1261] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8 [term 1 FOLLOWER]: Refusing update from remote peer 5fb9e57173d94423a7e31e876116de53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:06.143702 21408 raft_consensus.cc:1261] T d47f7cf6902a47708761e5da7a8231d6 P ae618bb3e59841e0a60be96d1b74235e [term 1 FOLLOWER]: Refusing update from remote peer 5fb9e57173d94423a7e31e876116de53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:06.143714 21819 consensus_queue.cc:1035] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c89c8d94bf79445189e99922cdefdef8" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 42869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:06.143927 21819 consensus_queue.cc:1035] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae618bb3e59841e0a60be96d1b74235e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:06.144426 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:06.144314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.130:59229)
0928 22:54:06.144351 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:06.144393 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 22:54:06.144933 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:06.144832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.129:50991)
0928 22:54:06.144876 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:06.144901 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 22:54:06.146013 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:06.145866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.131:57915)
0928 22:54:06.145955 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:06.145982 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 22:54:06.146324 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:06.146242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.129:35715)
0928 22:54:06.146277 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:06.146299 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 22:54:06.148067 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:06.147906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.14.14.130:59269)
0928 22:54:06.148006 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:06.148034 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 22:54:06.148132 21275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:54:06.148221 21275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:54:06.149477 21275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43f4101c5f22495b9950e8779f0fa717: Sending DeleteTablet for 3 replicas of tablet d47f7cf6902a47708761e5da7a8231d6
I20260928 22:54:06.149809 21388 tablet_service.cc:1514] Processing DeleteTablet for tablet d47f7cf6902a47708761e5da7a8231d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:06 UTC) from {username='oryx', principal='oryx/127.14.14.188@KRBTEST.COM'} at 127.0.0.1:38172
I20260928 22:54:06.149937 21525 tablet_service.cc:1514] Processing DeleteTablet for tablet d47f7cf6902a47708761e5da7a8231d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:06 UTC) from {username='oryx', principal='oryx/127.14.14.188@KRBTEST.COM'} at 127.0.0.1:45796
I20260928 22:54:06.150142 21840 tablet_replica.cc:330] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8: stopping tablet replica
I20260928 22:54:06.150200 21840 raft_consensus.cc:2229] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:54:06.150249 21840 raft_consensus.cc:2258] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:06.150517 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 21236
I20260928 22:54:06.150628 21840 ts_tablet_manager.cc:1905] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:06.152953 21840 ts_tablet_manager.cc:1918] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:54:06.153033 21840 log.cc:1196] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/d47f7cf6902a47708761e5da7a8231d6
I20260928 22:54:06.153267 21840 ts_tablet_manager.cc:1939] T d47f7cf6902a47708761e5da7a8231d6 P c89c8d94bf79445189e99922cdefdef8: Deleting consensus metadata
I20260928 22:54:06.153628 21260 catalog_manager.cc:4841] TS c89c8d94bf79445189e99922cdefdef8 (127.14.14.130:42869): tablet d47f7cf6902a47708761e5da7a8231d6 (table test-table [id=8f25b56cb833442e84ce6aa6727d4f41]) successfully deleted
I20260928 22:54:06.153993 21659 tablet_service.cc:1514] Processing DeleteTablet for tablet d47f7cf6902a47708761e5da7a8231d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:06 UTC) from {username='oryx', principal='oryx/127.14.14.188@KRBTEST.COM'} at 127.0.0.1:60982
I20260928 22:54:06.154181 21841 tablet_replica.cc:330] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53: stopping tablet replica
I20260928 22:54:06.154251 21841 raft_consensus.cc:2229] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:54:06.154342 21841 raft_consensus.cc:2258] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:06.154596 21841 ts_tablet_manager.cc:1905] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:06.156468 21841 ts_tablet_manager.cc:1918] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:54:06.156533 21841 log.cc:1196] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/d47f7cf6902a47708761e5da7a8231d6
I20260928 22:54:06.156770 21841 ts_tablet_manager.cc:1939] T d47f7cf6902a47708761e5da7a8231d6 P 5fb9e57173d94423a7e31e876116de53: Deleting consensus metadata
I20260928 22:54:06.157094 21260 catalog_manager.cc:4841] TS 5fb9e57173d94423a7e31e876116de53 (127.14.14.131:36527): tablet d47f7cf6902a47708761e5da7a8231d6 (table test-table [id=8f25b56cb833442e84ce6aa6727d4f41]) successfully deleted
W20260928 22:54:06.159417 21811 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:06.159264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:45887 (local address 127.0.0.1:38182)
0928 22:54:06.159327 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:06.159361 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:45887: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 22:54:06.159551 21259 catalog_manager.cc:4584] TS ae618bb3e59841e0a60be96d1b74235e (127.14.14.129:45887): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d47f7cf6902a47708761e5da7a8231d6: Network error: Client connection negotiation failed: client connection to 127.14.14.129:45887: connect: Connection reset by peer (error 104)
I20260928 22:54:06.159603 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 21462
I20260928 22:54:06.165372 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 21598
I20260928 22:54:06.170171 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 20955
I20260928 22:54:06.173512 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:06.173374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:44422)
0928 22:54:06.173435 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:06.173470 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 22:54:06.173915 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:06.173814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35807 (local address 127.0.0.1:44438)
0928 22:54:06.173855 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:06.173883 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 22:54:06.175081 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 21143
2026-09-28T22:54:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4943 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:54:06 dist-test-slave-kzfr krb5kdc[21844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:54:06 dist-test-slave-kzfr krb5kdc[21844](info): set up 2 sockets
Sep 28 22:54:06 dist-test-slave-kzfr krb5kdc[21844](info): commencing operation
krb5kdc: starting...
W20260928 22:54:07.266381 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:54:07 dist-test-slave-kzfr krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:54:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:54:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.190.keytab.
Entry for principal oryx/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.190.keytab.
I20260928 22:54:07.385159 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:38855
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.190.keytab
--principal=oryx/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:07.465055 21857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:07.465289 21857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:07.465313 21857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:07.466536 21857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:54:07.466580 21857 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 22:54:07.466593 21857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:07.466606 21857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:07.466619 21857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:54:07.466631 21857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:54:07.468068 21857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:38855
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.190.keytab
--principal=oryx/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:38855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:07.468252 21857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:07.468415 21857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:07.473336 21862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:07.473492 21857 server_base.cc:1034] running on GCE node
W20260928 22:54:07.473361 21863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:07.473361 21865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:07.473843 21857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:07.474159 21857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:07.475312 21857 hybrid_clock.cc:648] HybridClock initialized: now 1790636047475289 us; error 35 us; skew 500 ppm
Sep 28 22:54:07 dist-test-slave-kzfr krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636047, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:07.477852 21857 init.cc:377] Logged in from keytab as oryx/127.14.14.190@KRBTEST.COM (short username oryx)
I20260928 22:54:07.478282 21857 webserver.cc:460] Webserver started at http://127.14.14.190:46661/ using document root <none> and password file <none>
I20260928 22:54:07.478507 21857 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:07.478550 21857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:07.478721 21857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:07.479854 21857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "c1fdfd4743a84e26abd80f95a1ad1088"
format_stamp: "Formatted at 2026-09-28 22:54:07 on dist-test-slave-kzfr"
server_key: "58421def05d8c5c6867b5164e661353c"
server_key_iv: "97fd38c3a86bb09fa6220e5dded1c07d"
server_key_version: "encryptionkey@0"
I20260928 22:54:07.480208 21857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "c1fdfd4743a84e26abd80f95a1ad1088"
format_stamp: "Formatted at 2026-09-28 22:54:07 on dist-test-slave-kzfr"
server_key: "58421def05d8c5c6867b5164e661353c"
server_key_iv: "97fd38c3a86bb09fa6220e5dded1c07d"
server_key_version: "encryptionkey@0"
I20260928 22:54:07.481539 21857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:54:07.482402 21872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:07.482770 21857 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:54:07.482854 21857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "c1fdfd4743a84e26abd80f95a1ad1088"
format_stamp: "Formatted at 2026-09-28 22:54:07 on dist-test-slave-kzfr"
server_key: "58421def05d8c5c6867b5164e661353c"
server_key_iv: "97fd38c3a86bb09fa6220e5dded1c07d"
server_key_version: "encryptionkey@0"
I20260928 22:54:07.482959 21857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:07.500198 21857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:07.502504 21857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:07.502656 21857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:07.507622 21857 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:38855
I20260928 22:54:07.507671 21934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:38855 every 8 connection(s)
I20260928 22:54:07.507992 21857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:54:07.508689 21935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:07.509946 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 21857
I20260928 22:54:07.510043 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:54:07.510233 14394 external_mini_cluster.cc:1428] Setting key 726837c52ff2efecac517b4ecc4b1f16
I20260928 22:54:07.511443 21935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088: Bootstrap starting.
I20260928 22:54:07.512032 21935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:07.512324 21935 log.cc:824] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:07.513000 21935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088: No bootstrap required, opened a new log
I20260928 22:54:07.515053 21935 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1fdfd4743a84e26abd80f95a1ad1088" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38855 } }
I20260928 22:54:07.515215 21935 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:07.515264 21935 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1fdfd4743a84e26abd80f95a1ad1088, State: Initialized, Role: FOLLOWER
I20260928 22:54:07.515424 21935 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1fdfd4743a84e26abd80f95a1ad1088" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38855 } }
I20260928 22:54:07.515537 21935 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:54:07.515582 21935 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:54:07.515632 21935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:07.516140 21935 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1fdfd4743a84e26abd80f95a1ad1088" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38855 } }
I20260928 22:54:07.516265 21935 leader_election.cc:302] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1fdfd4743a84e26abd80f95a1ad1088; no voters:
I20260928 22:54:07.516438 21935 leader_election.cc:288] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:54:07.516620 21941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:54:07.516726 21935 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:54:07.516764 21941 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [term 1 LEADER]: Becoming Leader. State: Replica: c1fdfd4743a84e26abd80f95a1ad1088, State: Running, Role: LEADER
I20260928 22:54:07.516839 21941 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1fdfd4743a84e26abd80f95a1ad1088" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38855 } }
W20260928 22:54:07.518405 21955 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:54:07.518466 21955 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:54:07.518493 21940 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1fdfd4743a84e26abd80f95a1ad1088" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1fdfd4743a84e26abd80f95a1ad1088" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38855 } } }
I20260928 22:54:07.518554 21940 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [sys.catalog]: This master's current role is: LEADER
Sep 28 22:54:07 dist-test-slave-kzfr krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
I20260928 22:54:07.518764 21942 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1fdfd4743a84e26abd80f95a1ad1088. Latest consensus state: current_term: 1 leader_uuid: "c1fdfd4743a84e26abd80f95a1ad1088" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1fdfd4743a84e26abd80f95a1ad1088" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38855 } } }
I20260928 22:54:07.518764 21957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:54:07.518839 21942 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088 [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:07.518985 21957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:54:07.520551 21957 catalog_manager.cc:1347] Generated new cluster ID: 252aae321b4841449f1e293ffc440cdb
I20260928 22:54:07.520606 21957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:54:07.523219 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:07.510750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.14.14.190:38855)
0928 22:54:07.510939 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:07.510949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:07.512688 (+ 1739us) server_negotiation.cc:407] Connection header received
0928 22:54:07.513206 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:07.513213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:07.513268 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:07.513419 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:07.514103 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.514972 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.515940 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.516341 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.516454 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:07.519351 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:07.519355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:07.519355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:07.519367 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:07.521373 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:07.521796 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:07.521800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:07.521800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:07.521911 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:07.522129 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:07.522130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:07.522130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:07.522439 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:07.522600 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:07.522685 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 22:54:07.522822 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260928 22:54:07.546080 21957 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:54:07.546866 21957 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:54:07.561319 21957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088: Generated new TSK 0
I20260928 22:54:07.561558 21957 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.129.keytab.
Entry for principal oryx/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.129.keytab.
I20260928 22:54:07.594362 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38855
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.129.keytab
--principal=oryx/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:07.680030 21961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:07.680224 21961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:07.680248 21961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:07.681420 21961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:07.681463 21961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:07.681483 21961 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:54:07.681501 21961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:54:07.683163 21961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.129.keytab
--principal=oryx/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.14.190:38855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:07.683408 21961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:07.683665 21961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:07.688792 21969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:07.688794 21966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:07.688997 21967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:07.689157 21961 server_base.cc:1034] running on GCE node
I20260928 22:54:07.689401 21961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:07.689692 21961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:07.690860 21961 hybrid_clock.cc:648] HybridClock initialized: now 1790636047690829 us; error 43 us; skew 500 ppm
Sep 28 22:54:07 dist-test-slave-kzfr krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636047, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:07.693339 21961 init.cc:377] Logged in from keytab as oryx/127.14.14.129@KRBTEST.COM (short username oryx)
I20260928 22:54:07.693887 21961 webserver.cc:460] Webserver started at http://127.14.14.129:33503/ using document root <none> and password file <none>
I20260928 22:54:07.694131 21961 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:07.694198 21961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:07.694334 21961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:07.695364 21961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "bc4b59c4eea843f4a1d6fb1d33640071"
format_stamp: "Formatted at 2026-09-28 22:54:07 on dist-test-slave-kzfr"
server_key: "1cf9bbbcc14cce645d47a7293046e750"
server_key_iv: "3e20f56b3fbd6330ed6b187bc03971fe"
server_key_version: "encryptionkey@0"
I20260928 22:54:07.695736 21961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "bc4b59c4eea843f4a1d6fb1d33640071"
format_stamp: "Formatted at 2026-09-28 22:54:07 on dist-test-slave-kzfr"
server_key: "1cf9bbbcc14cce645d47a7293046e750"
server_key_iv: "3e20f56b3fbd6330ed6b187bc03971fe"
server_key_version: "encryptionkey@0"
I20260928 22:54:07.697068 21961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:54:07.697938 21976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:07.698128 21961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:07.698225 21961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "bc4b59c4eea843f4a1d6fb1d33640071"
format_stamp: "Formatted at 2026-09-28 22:54:07 on dist-test-slave-kzfr"
server_key: "1cf9bbbcc14cce645d47a7293046e750"
server_key_iv: "3e20f56b3fbd6330ed6b187bc03971fe"
server_key_version: "encryptionkey@0"
I20260928 22:54:07.698302 21961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:07.717551 21961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:07.719530 21961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:07.719692 21961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:07.720227 21961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:07.720289 21961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:07.720335 21961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:07.720376 21961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:54:07 dist-test-slave-kzfr krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636047, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
I20260928 22:54:07.727833 21961 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:44925
I20260928 22:54:07.728127 22091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:44925 every 8 connection(s)
I20260928 22:54:07.728317 21961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:54:07.730110 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 21961
I20260928 22:54:07.730234 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:54:07.730468 14394 external_mini_cluster.cc:1428] Setting key 36d39196eb66e44e776d8d031a6ccd7a
I20260928 22:54:07.730715 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:07.720793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:34397 (local address 127.14.14.190:38855)
0928 22:54:07.720887 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:07.720890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:07.721051 (+ 161us) server_negotiation.cc:407] Connection header received
0928 22:54:07.721547 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:07.721548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:07.721562 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:07.721661 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:07.722239 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.723097 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.724841 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.725220 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.725302 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:07.727586 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:07.727588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:07.727588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:07.727596 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:07.729246 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:07.729672 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:07.729673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:07.729673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:07.729734 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:07.729976 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:07.729977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:07.729977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:07.730137 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:07.730220 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:07.730494 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 22:54:07.730590 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 22:54:07.730856 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:07.720754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38855 (local address 127.14.14.129:34397)
0928 22:54:07.720916 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:07.720956 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:07.721448 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:07.721726 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:07.721730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:07.721959 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:07.722162 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:07.722166 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.723230 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.723231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:07.724728 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:07.724732 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.725333 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.725334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:07.725461 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:07.726079 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:07.726087 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:07.727484 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:07.729320 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:07.729326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:07.729327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:07.729561 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:07.729795 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:07.729796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:07.729796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:07.729912 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:07.730210 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:07.730216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:07.730365 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 22:54:07.730451 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 22:54:07.730634 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260928 22:54:07.731451 22092 heartbeater.cc:347] Connected to a master server at 127.14.14.190:38855
I20260928 22:54:07.731537 22092 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:07.731731 22092 heartbeater.cc:511] Master 127.14.14.190:38855 requested a full tablet report, sending...
I20260928 22:54:07.732317 21889 ts_manager.cc:194] Registered new tserver with Master: bc4b59c4eea843f4a1d6fb1d33640071 (127.14.14.129:44925)
I20260928 22:54:07.732925 21889 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:34397
I20260928 22:54:07.739775 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:07.733138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:42331 (local address 127.14.14.190:38855)
0928 22:54:07.733196 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:07.733198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:07.733302 (+ 104us) server_negotiation.cc:407] Connection header received
0928 22:54:07.733373 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:07.733374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:07.733385 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:07.733449 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:07.733927 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.734517 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.735674 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.735964 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.736025 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:07.737162 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:07.737163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:07.737163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:07.737169 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:07.738628 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:07.739051 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:07.739051 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:07.739052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:07.739101 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:07.739321 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:07.739322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:07.739322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:07.739425 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:07.739473 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:07.739659 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 22:54:07.739713 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:54:07.739872 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:07.733081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38855 (local address 127.14.14.129:42331)
0928 22:54:07.733224 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:07.733234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:07.733305 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:07.733483 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:07.733484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:07.733739 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:07.733833 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:07.733835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.734651 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.734652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:07.735568 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:07.735571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.736056 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.736057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:07.736139 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:07.736596 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:07.736602 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:07.737065 (+ 463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:07.738745 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:07.738746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:07.738747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:07.738972 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:07.739174 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:07.739175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:07.739175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:07.739255 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:07.739496 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:07.739497 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:07.739552 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 22:54:07.739642 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 22:54:07.739750 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for oryx/127.14.14.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.130.keytab.
Entry for principal oryx/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.130.keytab.
I20260928 22:54:07.759068 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38855
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.130.keytab
--principal=oryx/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:07.845286 22101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:07.845546 22101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:07.845587 22101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:07.846761 22101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:07.846837 22101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:07.846872 22101 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:54:07.846905 22101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:54:07.848420 22101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.130.keytab
--principal=oryx/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.14.190:38855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:07.848685 22101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:07.848917 22101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:07.854229 22109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:07.854285 22107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:07.854261 22106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:07.854501 22101 server_base.cc:1034] running on GCE node
I20260928 22:54:07.854704 22101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:07.854940 22101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:07.856098 22101 hybrid_clock.cc:648] HybridClock initialized: now 1790636047856088 us; error 27 us; skew 500 ppm
Sep 28 22:54:07 dist-test-slave-kzfr krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636047, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:07.858687 22101 init.cc:377] Logged in from keytab as oryx/127.14.14.130@KRBTEST.COM (short username oryx)
I20260928 22:54:07.859124 22101 webserver.cc:460] Webserver started at http://127.14.14.130:35267/ using document root <none> and password file <none>
I20260928 22:54:07.859349 22101 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:07.859419 22101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:07.859573 22101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:07.860661 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "477dbaca260f4a329431eeec15f4d8ac"
format_stamp: "Formatted at 2026-09-28 22:54:07 on dist-test-slave-kzfr"
server_key: "ec58ba9d7ef4c23593995b51400877ef"
server_key_iv: "d248e4940ac1f32ee93058932c7818ba"
server_key_version: "encryptionkey@0"
I20260928 22:54:07.861049 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "477dbaca260f4a329431eeec15f4d8ac"
format_stamp: "Formatted at 2026-09-28 22:54:07 on dist-test-slave-kzfr"
server_key: "ec58ba9d7ef4c23593995b51400877ef"
server_key_iv: "d248e4940ac1f32ee93058932c7818ba"
server_key_version: "encryptionkey@0"
I20260928 22:54:07.862536 22101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:54:07.863289 22116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:07.863478 22101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:07.863586 22101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "477dbaca260f4a329431eeec15f4d8ac"
format_stamp: "Formatted at 2026-09-28 22:54:07 on dist-test-slave-kzfr"
server_key: "ec58ba9d7ef4c23593995b51400877ef"
server_key_iv: "d248e4940ac1f32ee93058932c7818ba"
server_key_version: "encryptionkey@0"
I20260928 22:54:07.863673 22101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:07.879500 22101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:07.881897 22101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:07.882066 22101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:07.882568 22101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:07.882637 22101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:07.882681 22101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:07.882704 22101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:07.890019 22101 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:37249
I20260928 22:54:07.890082 22231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:37249 every 8 connection(s)
Sep 28 22:54:07 dist-test-slave-kzfr krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636047, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
I20260928 22:54:07.890614 22101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:54:07.893463 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:07.883123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:49491 (local address 127.14.14.190:38855)
0928 22:54:07.883167 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:07.883169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:07.883377 (+ 208us) server_negotiation.cc:407] Connection header received
0928 22:54:07.884013 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:07.884015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:07.884035 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:07.884100 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:07.884974 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.885894 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.887384 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.887740 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.887827 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:07.890919 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:07.890921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:07.890922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:07.890931 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:07.892290 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:07.892659 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:07.892660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:07.892660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:07.892709 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:07.892913 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:07.892914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:07.892914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:07.893057 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:07.893109 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:07.893336 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 22:54:07.893395 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:54:07.893683 22125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:07.883109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38855 (local address 127.14.14.130:49491)
0928 22:54:07.883252 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:07.883285 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:07.883807 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:07.884194 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:07.884199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:07.884568 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:07.884861 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:07.884868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.886015 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.886016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:07.887249 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:07.887252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.887839 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.887840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:07.887970 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:07.888650 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:07.888662 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:07.890794 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:07.892363 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:07.892368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:07.892369 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:07.892594 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:07.892767 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:07.892768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:07.892768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:07.892862 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:07.893114 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:07.893118 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:07.893229 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 22:54:07.893310 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 22:54:07.893428 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260928 22:54:07.894182 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 22101
I20260928 22:54:07.894214 22232 heartbeater.cc:347] Connected to a master server at 127.14.14.190:38855
I20260928 22:54:07.894292 22232 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:07.894301 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:54:07.894481 22232 heartbeater.cc:511] Master 127.14.14.190:38855 requested a full tablet report, sending...
I20260928 22:54:07.894516 14394 external_mini_cluster.cc:1428] Setting key c67290b754dee81fb9b3717b6a225dc5
I20260928 22:54:07.894963 21888 ts_manager.cc:194] Registered new tserver with Master: 477dbaca260f4a329431eeec15f4d8ac (127.14.14.130:37249)
I20260928 22:54:07.895494 21888 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:49491
I20260928 22:54:07.901338 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:07.895311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:41853 (local address 127.14.14.190:38855)
0928 22:54:07.895348 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:07.895350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:07.895477 (+ 127us) server_negotiation.cc:407] Connection header received
0928 22:54:07.895544 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:07.895544 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:07.895555 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:07.895602 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:07.895988 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.896463 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.897516 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.897784 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.897842 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:07.898797 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:07.898798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:07.898799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:07.898804 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:07.900158 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:07.900587 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:07.900588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:07.900588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:07.900635 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:07.900857 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:07.900858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:07.900858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:07.900986 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:07.901041 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:07.901225 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 22:54:07.901282 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:54:07.901427 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:07.895237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38855 (local address 127.14.14.130:41853)
0928 22:54:07.895403 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:07.895412 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:07.895485 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:07.895665 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:07.895666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:07.895823 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:07.895892 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:07.895893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.896561 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.896562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:07.897428 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:07.897430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:07.897844 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:07.897845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:07.897915 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:07.898274 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:07.898279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:07.898714 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:07.900244 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:07.900245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:07.900246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:07.900503 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:07.900703 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:07.900704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:07.900704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:07.900775 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:07.901059 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:07.901060 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:07.901118 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 22:54:07.901198 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 22:54:07.901306 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":3,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for oryx/127.14.14.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.131.keytab.
Entry for principal oryx/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.131.keytab.
I20260928 22:54:07.920953 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38855
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.131.keytab
--principal=oryx/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:08.003288 22241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:08.003484 22241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:08.003508 22241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:08.004599 22241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:08.004637 22241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:08.004657 22241 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:54:08.004675 22241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:54:08.006213 22241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.131.keytab
--principal=oryx/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.14.190:38855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:08.006397 22241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:08.006574 22241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:08.011682 22247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:08.011647 22246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:08.011863 22241 server_base.cc:1034] running on GCE node
W20260928 22:54:08.012007 22249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:08.012253 22241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:08.012534 22241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:08.013655 22241 hybrid_clock.cc:648] HybridClock initialized: now 1790636048013633 us; error 31 us; skew 500 ppm
Sep 28 22:54:08 dist-test-slave-kzfr krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636048, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:08.016098 22241 init.cc:377] Logged in from keytab as oryx/127.14.14.131@KRBTEST.COM (short username oryx)
I20260928 22:54:08.016595 22241 webserver.cc:460] Webserver started at http://127.14.14.131:43361/ using document root <none> and password file <none>
I20260928 22:54:08.016882 22241 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:08.016940 22241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:08.017030 22241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:08.018242 22241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "820a659649bf412b99b65c5f2629e576"
format_stamp: "Formatted at 2026-09-28 22:54:08 on dist-test-slave-kzfr"
server_key: "fed68ec1d954d857fa22f3dc77def178"
server_key_iv: "881c31ed00689bd7534ce84bf3cf43c9"
server_key_version: "encryptionkey@0"
I20260928 22:54:08.018559 22241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "820a659649bf412b99b65c5f2629e576"
format_stamp: "Formatted at 2026-09-28 22:54:08 on dist-test-slave-kzfr"
server_key: "fed68ec1d954d857fa22f3dc77def178"
server_key_iv: "881c31ed00689bd7534ce84bf3cf43c9"
server_key_version: "encryptionkey@0"
I20260928 22:54:08.019891 22241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:54:08.020593 22256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:08.020845 22241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:08.020915 22241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "820a659649bf412b99b65c5f2629e576"
format_stamp: "Formatted at 2026-09-28 22:54:08 on dist-test-slave-kzfr"
server_key: "fed68ec1d954d857fa22f3dc77def178"
server_key_iv: "881c31ed00689bd7534ce84bf3cf43c9"
server_key_version: "encryptionkey@0"
I20260928 22:54:08.020960 22241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:08.036759 22241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:08.039222 22241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:08.039341 22241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:08.039825 22241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:08.039870 22241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:08.039896 22241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:08.039950 22241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:54:08 dist-test-slave-kzfr krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636048, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
I20260928 22:54:08.047875 22241 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:37017
I20260928 22:54:08.048061 22371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:37017 every 8 connection(s)
I20260928 22:54:08.048338 22241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:54:08.050618 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.040473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:53777 (local address 127.14.14.190:38855)
0928 22:54:08.040533 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.040535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.040744 (+ 209us) server_negotiation.cc:407] Connection header received
0928 22:54:08.041312 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.041314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.041327 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.041385 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:08.042234 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.043153 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.044294 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.044497 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.044585 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.047546 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:08.047547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:08.047548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:08.047557 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:08.049299 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:08.049715 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:08.049716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:08.049716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:08.049772 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:08.050030 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:08.050031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:08.050031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:08.050138 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:08.050190 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.050468 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.050551 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 22:54:08.050693 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.040460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38855 (local address 127.14.14.131:53777)
0928 22:54:08.040615 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.040650 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.041213 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.041453 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.041457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.041822 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:08.042109 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.042116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.043265 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.043266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.044193 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.044196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.044608 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.044609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.044735 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.045310 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:08.045319 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:08.047426 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:08.049374 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:08.049378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:08.049380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:08.049640 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:08.049829 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:08.049830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:08.049830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:08.049959 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:08.050194 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:08.050198 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:08.050322 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.050396 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.050510 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260928 22:54:08.052199 22372 heartbeater.cc:347] Connected to a master server at 127.14.14.190:38855
I20260928 22:54:08.052289 22372 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:08.052474 22372 heartbeater.cc:511] Master 127.14.14.190:38855 requested a full tablet report, sending...
I20260928 22:54:08.052871 21888 ts_manager.cc:194] Registered new tserver with Master: 820a659649bf412b99b65c5f2629e576 (127.14.14.131:37017)
I20260928 22:54:08.053308 21888 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:53777
I20260928 22:54:08.056143 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 22241
I20260928 22:54:08.056263 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:54:08.056442 14394 external_mini_cluster.cc:1428] Setting key d4fca4ebf37ef27dd008d9f65df4db52
I20260928 22:54:08.057195 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:54:08.058038 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.052262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:52577 (local address 127.14.14.190:38855)
0928 22:54:08.052303 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.052305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.052380 (+ 75us) server_negotiation.cc:407] Connection header received
0928 22:54:08.052465 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.052465 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.052473 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.052550 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:08.052917 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.053475 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.054238 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.054416 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.054478 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.055414 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:08.055415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:08.055415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:08.055421 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:08.056842 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:08.057225 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:08.057226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:08.057226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:08.057308 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:08.057514 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:08.057515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:08.057515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:08.057678 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:08.057735 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.057922 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.057978 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:54:08.058148 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.052153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38855 (local address 127.14.14.131:52577)
0928 22:54:08.052297 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.052308 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.052401 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.052557 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.052558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.052743 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:08.052819 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.052821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.053619 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.053620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.054165 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.054166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.054546 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.054547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.054601 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.055053 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:08.055058 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:08.055337 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:08.056920 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:08.056921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:08.056921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:08.057152 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:08.057387 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:08.057388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:08.057388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:08.057453 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:08.057750 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:08.057751 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:08.057817 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.057897 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.058004 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
Sep 28 22:54:08 dist-test-slave-kzfr krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 22:54:08 dist-test-slave-kzfr krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:08 dist-test-slave-kzfr krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 22:54:08.078840 21938 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:08.073478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.14.14.190:38855)
0928 22:54:08.073559 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.073562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.073625 (+ 63us) server_negotiation.cc:407] Connection header received
0928 22:54:08.073654 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.073655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.073663 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.073718 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:08.074289 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.074773 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.075514 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.075695 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.075752 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.078719 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54494: BlockingRecv error: recv got EOF from 127.0.0.1:54494 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 22:54:08 dist-test-slave-kzfr krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
I20260928 22:54:08.086499 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.079678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.14.14.190:38855)
0928 22:54:08.079751 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.079754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.079837 (+ 83us) server_negotiation.cc:407] Connection header received
0928 22:54:08.079912 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.079913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.079923 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.079971 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:08.080425 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.080897 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.081639 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.081835 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.081922 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.084021 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:08.084022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:08.084022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:08.084037 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:08.085406 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:08.085831 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:08.085831 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:08.085832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:08.085882 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:08.086117 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:08.086118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:08.086118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:08.086206 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:08.086276 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.086385 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.086438 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 22:54:08.088052 21888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:54:08.089686 21888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:54:08.096187 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.091838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:44925 (local address 127.0.0.1:36892)
0928 22:54:08.092010 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.092048 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.092106 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.092350 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.092353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.092355 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:08.092419 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.092420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.093763 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.093764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.095000 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.095004 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.095853 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.095854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.095939 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.095954 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.096046 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.096106 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260928 22:54:08.096202 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.091958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.14.14.129:44925)
0928 22:54:08.092108 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.092113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.092124 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:54:08.092223 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.092226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.092243 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.092329 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:08.092537 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.093682 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.095073 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.095759 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.095834 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.095909 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.096063 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.096102 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260928 22:54:08.096614 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.091838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37017 (local address 127.0.0.1:36146)
0928 22:54:08.092233 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.092244 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.092297 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.092714 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.092714 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.092717 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:08.092762 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.092763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.093990 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.093991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.095244 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.095247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.096162 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.096163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.096224 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.096228 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.096493 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.096543 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":39,"thread_start_us":75,"threads_started":1}
I20260928 22:54:08.096649 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.092246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36146 (local address 127.14.14.131:37017)
0928 22:54:08.092547 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.092552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.092561 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:54:08.092619 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.092621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.092638 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.092714 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:08.092860 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.093908 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.095350 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.095991 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.096034 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.096104 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.096513 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.096553 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":101,"threads_started":1}
I20260928 22:54:08.096987 22026 tablet_service.cc:1467] Processing CreateTablet for tablet 9108fea924b145f8b776787cac31fa5d (DEFAULT_TABLE table=test-table [id=519256a4eeab48989143931daef782f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:08.097364 22026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9108fea924b145f8b776787cac31fa5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:08.097396 22306 tablet_service.cc:1467] Processing CreateTablet for tablet 9108fea924b145f8b776787cac31fa5d (DEFAULT_TABLE table=test-table [id=519256a4eeab48989143931daef782f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:08.097756 22306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9108fea924b145f8b776787cac31fa5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:08.101404 22400 tablet_bootstrap.cc:492] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576: Bootstrap starting.
I20260928 22:54:08.101714 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.092101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:37249 (local address 127.0.0.1:50508)
0928 22:54:08.092797 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.092806 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.092849 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.093118 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.093119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.093121 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:08.093159 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.093160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.094307 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.094308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.095430 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.095432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.096931 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.096932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.097007 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.097012 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.101532 (+ 4520us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.101623 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260928 22:54:08.102043 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.092557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50508 (local address 127.14.14.130:37249)
0928 22:54:08.092980 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.092984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.092991 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:54:08.093020 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.093021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.093032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.093122 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:08.093231 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.094227 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.095506 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.096243 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.096597 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.096672 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.101866 (+ 5194us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.101918 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":88,"threads_started":1}
I20260928 22:54:08.102099 22401 tablet_bootstrap.cc:492] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071: Bootstrap starting.
I20260928 22:54:08.102160 22400 tablet_bootstrap.cc:654] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:08.102459 22400 log.cc:824] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:08.102705 22401 tablet_bootstrap.cc:654] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:08.102653 22166 tablet_service.cc:1467] Processing CreateTablet for tablet 9108fea924b145f8b776787cac31fa5d (DEFAULT_TABLE table=test-table [id=519256a4eeab48989143931daef782f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:08.103005 22166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9108fea924b145f8b776787cac31fa5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:08.103017 22401 log.cc:824] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:08.103348 22400 tablet_bootstrap.cc:492] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576: No bootstrap required, opened a new log
I20260928 22:54:08.103425 22400 ts_tablet_manager.cc:1397] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:54:08.103772 22401 tablet_bootstrap.cc:492] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071: No bootstrap required, opened a new log
I20260928 22:54:08.103842 22401 ts_tablet_manager.cc:1397] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:54:08.105063 22400 raft_consensus.cc:348] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.105217 22400 raft_consensus.cc:374] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:08.105260 22400 raft_consensus.cc:729] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820a659649bf412b99b65c5f2629e576, State: Initialized, Role: FOLLOWER
I20260928 22:54:08.105392 22400 consensus_queue.cc:260] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.105715 22400 ts_tablet_manager.cc:1428] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:54:08.105731 22372 heartbeater.cc:503] Master 127.14.14.190:38855 was elected leader, sending a full tablet report...
I20260928 22:54:08.106300 22401 raft_consensus.cc:348] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.106463 22401 raft_consensus.cc:374] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:08.106510 22401 raft_consensus.cc:729] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4b59c4eea843f4a1d6fb1d33640071, State: Initialized, Role: FOLLOWER
I20260928 22:54:08.106659 22401 consensus_queue.cc:260] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.106932 22401 ts_tablet_manager.cc:1428] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:54:08.106969 22092 heartbeater.cc:503] Master 127.14.14.190:38855 was elected leader, sending a full tablet report...
I20260928 22:54:08.107210 22406 tablet_bootstrap.cc:492] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac: Bootstrap starting.
I20260928 22:54:08.107798 22406 tablet_bootstrap.cc:654] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:08.108121 22406 log.cc:824] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:08.108767 22406 tablet_bootstrap.cc:492] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac: No bootstrap required, opened a new log
I20260928 22:54:08.108819 22406 ts_tablet_manager.cc:1397] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:54:08.110318 22406 raft_consensus.cc:348] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.110440 22406 raft_consensus.cc:374] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:08.110462 22406 raft_consensus.cc:729] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477dbaca260f4a329431eeec15f4d8ac, State: Initialized, Role: FOLLOWER
I20260928 22:54:08.110589 22406 consensus_queue.cc:260] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.110831 22406 ts_tablet_manager.cc:1428] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:54:08.110886 22232 heartbeater.cc:503] Master 127.14.14.190:38855 was elected leader, sending a full tablet report...
W20260928 22:54:08.141706 22233 tablet.cc:2369] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:54:08.229799 22093 tablet.cc:2369] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:54:08.231328 22408 raft_consensus.cc:482] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:54:08.231443 22408 raft_consensus.cc:504] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.231761 22408 leader_election.cc:288] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc4b59c4eea843f4a1d6fb1d33640071 (127.14.14.129:44925), 820a659649bf412b99b65c5f2629e576 (127.14.14.131:37017)
I20260928 22:54:08.234920 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.232047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:44021 (local address 127.14.14.131:37017)
0928 22:54:08.232090 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.232092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.232244 (+ 152us) server_negotiation.cc:407] Connection header received
0928 22:54:08.232352 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.232353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.232363 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.232432 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:08.232630 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.233076 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.234140 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.234543 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.234599 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.234606 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.234798 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.234846 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:54:08.234989 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.232037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:58707 (local address 127.14.14.129:44925)
0928 22:54:08.232082 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.232084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.232097 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:54:08.232185 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.232186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.232197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.232262 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:08.232474 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.233074 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.234196 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.234561 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.234658 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.234666 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.234860 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.234904 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:54:08.234956 22125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.231966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:44925 (local address 127.14.14.130:58707)
0928 22:54:08.232012 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.232024 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.232091 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.232273 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.232273 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.232275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:08.232372 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.232374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.233169 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.233170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.234081 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.234083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.234670 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.234671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.234771 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.234775 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.234834 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.234873 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":15}
I20260928 22:54:08.234933 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.231966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37017 (local address 127.14.14.130:44021)
0928 22:54:08.232170 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.232180 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.232253 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.232480 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.232481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.232483 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:08.232556 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.232558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.233170 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.233171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.234049 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.234051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.234627 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.234628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.234722 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.234726 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.234772 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.234833 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":79,"thread_start_us":89,"threads_started":1}
I20260928 22:54:08.235370 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108fea924b145f8b776787cac31fa5d" candidate_uuid: "477dbaca260f4a329431eeec15f4d8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" is_pre_election: true
I20260928 22:54:08.235430 22326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108fea924b145f8b776787cac31fa5d" candidate_uuid: "477dbaca260f4a329431eeec15f4d8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820a659649bf412b99b65c5f2629e576" is_pre_election: true
I20260928 22:54:08.235548 22046 raft_consensus.cc:2454] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 477dbaca260f4a329431eeec15f4d8ac in term 0.
I20260928 22:54:08.235546 22326 raft_consensus.cc:2454] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 477dbaca260f4a329431eeec15f4d8ac in term 0.
I20260928 22:54:08.236060 22120 leader_election.cc:302] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 477dbaca260f4a329431eeec15f4d8ac, bc4b59c4eea843f4a1d6fb1d33640071; no voters:
I20260928 22:54:08.236219 22408 raft_consensus.cc:2781] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:54:08.236277 22408 raft_consensus.cc:482] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:54:08.236310 22408 raft_consensus.cc:3037] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:08.237030 22408 raft_consensus.cc:504] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.237181 22408 leader_election.cc:288] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [CANDIDATE]: Term 1 election: Requested vote from peers bc4b59c4eea843f4a1d6fb1d33640071 (127.14.14.129:44925), 820a659649bf412b99b65c5f2629e576 (127.14.14.131:37017)
I20260928 22:54:08.237326 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108fea924b145f8b776787cac31fa5d" candidate_uuid: "477dbaca260f4a329431eeec15f4d8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc4b59c4eea843f4a1d6fb1d33640071"
I20260928 22:54:08.237329 22326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108fea924b145f8b776787cac31fa5d" candidate_uuid: "477dbaca260f4a329431eeec15f4d8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820a659649bf412b99b65c5f2629e576"
I20260928 22:54:08.237399 22326 raft_consensus.cc:3037] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:08.237423 22046 raft_consensus.cc:3037] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:08.238166 22326 raft_consensus.cc:2454] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 477dbaca260f4a329431eeec15f4d8ac in term 1.
I20260928 22:54:08.238286 22046 raft_consensus.cc:2454] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 477dbaca260f4a329431eeec15f4d8ac in term 1.
I20260928 22:54:08.238364 22118 leader_election.cc:302] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 477dbaca260f4a329431eeec15f4d8ac, 820a659649bf412b99b65c5f2629e576; no voters:
I20260928 22:54:08.238449 22408 raft_consensus.cc:2781] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:54:08.238590 22408 raft_consensus.cc:686] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [term 1 LEADER]: Becoming Leader. State: Replica: 477dbaca260f4a329431eeec15f4d8ac, State: Running, Role: LEADER
I20260928 22:54:08.238683 22408 consensus_queue.cc:237] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.239527 21888 catalog_manager.cc:5495] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 477dbaca260f4a329431eeec15f4d8ac (127.14.14.130). New cstate: current_term: 1 leader_uuid: "477dbaca260f4a329431eeec15f4d8ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } health_report { overall_health: HEALTHY } } }
I20260928 22:54:08.244547 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.242526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.14.14.130:37249)
0928 22:54:08.242564 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.242566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.242620 (+ 54us) server_negotiation.cc:407] Connection header received
0928 22:54:08.242678 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.242678 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.242692 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.242750 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:54:08.242858 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.243288 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.243984 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.244154 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.244203 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.244325 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:54:08.244405 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:54:08.244458 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.244489 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.244503 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:54:08.246101 22046 raft_consensus.cc:1261] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Refusing update from remote peer 477dbaca260f4a329431eeec15f4d8ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:08.246135 22326 raft_consensus.cc:1261] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Refusing update from remote peer 477dbaca260f4a329431eeec15f4d8ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:08.246418 22408 consensus_queue.cc:1035] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:08.246564 22410 consensus_queue.cc:1035] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:08.248870 22413 mvcc.cc:205] Tried to move back new op lower bound from 7334445253613584384 to 7334445253586161664. Current Snapshot: MvccSnapshot[applied={T|T < 7334445253613584384}]
I20260928 22:54:08.249768 22414 mvcc.cc:205] Tried to move back new op lower bound from 7334445253613584384 to 7334445253586161664. Current Snapshot: MvccSnapshot[applied={T|T < 7334445253613584384 or (T in {7334445253613584384})}]
I20260928 22:54:08.250890 21889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:54:08.250972 21889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:54:08.251696 21889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088: Sending DeleteTablet for 3 replicas of tablet 9108fea924b145f8b776787cac31fa5d
I20260928 22:54:08.251991 22026 tablet_service.cc:1514] Processing DeleteTablet for tablet 9108fea924b145f8b776787cac31fa5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:08 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:36892
I20260928 22:54:08.252110 22306 tablet_service.cc:1514] Processing DeleteTablet for tablet 9108fea924b145f8b776787cac31fa5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:08 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:36146
I20260928 22:54:08.252216 22423 tablet_replica.cc:330] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071: stopping tablet replica
I20260928 22:54:08.252216 22166 tablet_service.cc:1514] Processing DeleteTablet for tablet 9108fea924b145f8b776787cac31fa5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:08 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:50508
I20260928 22:54:08.252310 22423 raft_consensus.cc:2229] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:54:08.252406 22423 raft_consensus.cc:2258] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:08.252424 21889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:54:08.252561 21889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:54:08.252758 22424 tablet_replica.cc:330] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576: stopping tablet replica
I20260928 22:54:08.252846 22424 raft_consensus.cc:2229] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:54:08.253288 22423 ts_tablet_manager.cc:1905] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:08.253441 22424 raft_consensus.cc:2258] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:08.253702 22026 tablet_service.cc:1467] Processing CreateTablet for tablet 0faebe57e3db43eb97340af7b701aca4 (DEFAULT_TABLE table=test-table [id=1a20435bf1c6426195e0a82dee13d8df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:08.253768 22166 tablet_service.cc:1467] Processing CreateTablet for tablet 0faebe57e3db43eb97340af7b701aca4 (DEFAULT_TABLE table=test-table [id=1a20435bf1c6426195e0a82dee13d8df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:08.253821 22026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faebe57e3db43eb97340af7b701aca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:08.253886 22166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faebe57e3db43eb97340af7b701aca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:08.255126 22401 tablet_bootstrap.cc:492] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071: Bootstrap starting.
I20260928 22:54:08.255239 22406 tablet_bootstrap.cc:492] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac: Bootstrap starting.
I20260928 22:54:08.255553 22401 tablet_bootstrap.cc:654] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:08.255620 22406 tablet_bootstrap.cc:654] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:08.256130 22423 ts_tablet_manager.cc:1918] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:54:08.256227 22401 tablet_bootstrap.cc:492] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071: No bootstrap required, opened a new log
I20260928 22:54:08.256250 22423 log.cc:1196] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/wal/wals/9108fea924b145f8b776787cac31fa5d
I20260928 22:54:08.256319 22306 tablet_service.cc:1467] Processing CreateTablet for tablet 0faebe57e3db43eb97340af7b701aca4 (DEFAULT_TABLE table=test-table [id=1a20435bf1c6426195e0a82dee13d8df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:08.256420 22306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faebe57e3db43eb97340af7b701aca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:08.256282 22401 ts_tablet_manager.cc:1397] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:08.256580 22401 raft_consensus.cc:348] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.256661 22401 raft_consensus.cc:374] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:08.256688 22401 raft_consensus.cc:729] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4b59c4eea843f4a1d6fb1d33640071, State: Initialized, Role: FOLLOWER
I20260928 22:54:08.256735 22401 consensus_queue.cc:260] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.256852 22401 ts_tablet_manager.cc:1428] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:08.257261 22423 ts_tablet_manager.cc:1939] T 9108fea924b145f8b776787cac31fa5d P bc4b59c4eea843f4a1d6fb1d33640071: Deleting consensus metadata
I20260928 22:54:08.257727 21876 catalog_manager.cc:4841] TS bc4b59c4eea843f4a1d6fb1d33640071 (127.14.14.129:44925): tablet 9108fea924b145f8b776787cac31fa5d (table test-table [id=519256a4eeab48989143931daef782f3]) successfully deleted
I20260928 22:54:08.258073 22424 ts_tablet_manager.cc:1905] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:08.258177 22406 tablet_bootstrap.cc:492] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac: No bootstrap required, opened a new log
I20260928 22:54:08.258221 22406 ts_tablet_manager.cc:1397] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 22:54:08.258235 22400 tablet_bootstrap.cc:492] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576: Bootstrap starting.
I20260928 22:54:08.258358 22406 raft_consensus.cc:348] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.258423 22406 raft_consensus.cc:374] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:08.258460 22406 raft_consensus.cc:729] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477dbaca260f4a329431eeec15f4d8ac, State: Initialized, Role: FOLLOWER
I20260928 22:54:08.258503 22406 consensus_queue.cc:260] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.258594 22406 ts_tablet_manager.cc:1428] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:08.258620 22400 tablet_bootstrap.cc:654] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:08.259246 22400 tablet_bootstrap.cc:492] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576: No bootstrap required, opened a new log
I20260928 22:54:08.259301 22400 ts_tablet_manager.cc:1397] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:08.259433 22400 raft_consensus.cc:348] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.259516 22400 raft_consensus.cc:374] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:08.259539 22400 raft_consensus.cc:729] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820a659649bf412b99b65c5f2629e576, State: Initialized, Role: FOLLOWER
I20260928 22:54:08.259642 22425 tablet_replica.cc:330] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac: stopping tablet replica
I20260928 22:54:08.259645 22400 consensus_queue.cc:260] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.259734 22425 raft_consensus.cc:2229] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:54:08.259752 22400 ts_tablet_manager.cc:1428] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:08.259828 22425 raft_consensus.cc:2258] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:08.260226 22425 ts_tablet_manager.cc:1905] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:08.261714 22424 ts_tablet_manager.cc:1918] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:54:08.261711 22425 ts_tablet_manager.cc:1918] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:54:08.261793 22425 log.cc:1196] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/9108fea924b145f8b776787cac31fa5d
I20260928 22:54:08.261803 22424 log.cc:1196] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/9108fea924b145f8b776787cac31fa5d
I20260928 22:54:08.262046 22424 ts_tablet_manager.cc:1939] T 9108fea924b145f8b776787cac31fa5d P 820a659649bf412b99b65c5f2629e576: Deleting consensus metadata
I20260928 22:54:08.262061 22425 ts_tablet_manager.cc:1939] T 9108fea924b145f8b776787cac31fa5d P 477dbaca260f4a329431eeec15f4d8ac: Deleting consensus metadata
I20260928 22:54:08.262420 21874 catalog_manager.cc:4841] TS 820a659649bf412b99b65c5f2629e576 (127.14.14.131:37017): tablet 9108fea924b145f8b776787cac31fa5d (table test-table [id=519256a4eeab48989143931daef782f3]) successfully deleted
I20260928 22:54:08.262602 21874 catalog_manager.cc:4841] TS 477dbaca260f4a329431eeec15f4d8ac (127.14.14.130:37249): tablet 9108fea924b145f8b776787cac31fa5d (table test-table [id=519256a4eeab48989143931daef782f3]) successfully deleted
W20260928 22:54:08.299062 22373 tablet.cc:2369] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:54:08.368068 22404 raft_consensus.cc:482] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:54:08.368171 22404 raft_consensus.cc:504] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.368525 22404 leader_election.cc:288] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc4b59c4eea843f4a1d6fb1d33640071 (127.14.14.129:44925), 477dbaca260f4a329431eeec15f4d8ac (127.14.14.130:37249)
I20260928 22:54:08.372398 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.368900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:54775 (local address 127.14.14.129:44925)
0928 22:54:08.368943 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.368945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.369005 (+ 60us) server_negotiation.cc:407] Connection header received
0928 22:54:08.369033 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.369033 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.369043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.369096 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:08.369347 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.369913 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.371563 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.372024 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.372100 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.372110 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.372297 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.372339 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:54:08.372390 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.368719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:44925 (local address 127.14.14.131:54775)
0928 22:54:08.368830 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.368841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.368900 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.369143 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.369143 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.369145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:08.369255 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.369257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.370144 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.370145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.371456 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.371459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.372109 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.372110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.372196 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.372201 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.372254 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.372311 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":66}
I20260928 22:54:08.372702 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.368780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:44273 (local address 127.14.14.130:37249)
0928 22:54:08.368824 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.368827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.368925 (+ 98us) server_negotiation.cc:407] Connection header received
0928 22:54:08.369033 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.369034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.369045 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.369075 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:08.369356 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.370109 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.371583 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.372160 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.372240 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.372249 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.372595 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.372636 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:54:08.372670 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faebe57e3db43eb97340af7b701aca4" candidate_uuid: "820a659649bf412b99b65c5f2629e576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" is_pre_election: true
I20260928 22:54:08.372771 22046 raft_consensus.cc:2454] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 820a659649bf412b99b65c5f2629e576 in term 0.
I20260928 22:54:08.372927 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.368724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:37249 (local address 127.14.14.131:44273)
0928 22:54:08.368859 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.368871 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.368953 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.369143 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.369143 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.369145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:08.369255 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.369257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.370194 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.370194 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.371459 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.371460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.372286 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.372288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.372363 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.372367 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.372794 (+ 427us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.372840 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":21,"thread_start_us":57,"threads_started":1}
I20260928 22:54:08.373015 22260 leader_election.cc:302] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 820a659649bf412b99b65c5f2629e576, bc4b59c4eea843f4a1d6fb1d33640071; no voters:
I20260928 22:54:08.373140 22404 raft_consensus.cc:2781] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:54:08.373167 22404 raft_consensus.cc:482] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:54:08.373195 22404 raft_consensus.cc:3037] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:08.373193 22186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faebe57e3db43eb97340af7b701aca4" candidate_uuid: "820a659649bf412b99b65c5f2629e576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477dbaca260f4a329431eeec15f4d8ac" is_pre_election: true
I20260928 22:54:08.373312 22186 raft_consensus.cc:2454] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 820a659649bf412b99b65c5f2629e576 in term 0.
I20260928 22:54:08.374115 22404 raft_consensus.cc:504] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.374275 22404 leader_election.cc:288] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [CANDIDATE]: Term 1 election: Requested vote from peers bc4b59c4eea843f4a1d6fb1d33640071 (127.14.14.129:44925), 477dbaca260f4a329431eeec15f4d8ac (127.14.14.130:37249)
I20260928 22:54:08.374467 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faebe57e3db43eb97340af7b701aca4" candidate_uuid: "820a659649bf412b99b65c5f2629e576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc4b59c4eea843f4a1d6fb1d33640071"
I20260928 22:54:08.374552 22046 raft_consensus.cc:3037] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:08.375249 22046 raft_consensus.cc:2454] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 820a659649bf412b99b65c5f2629e576 in term 1.
I20260928 22:54:08.375339 22186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faebe57e3db43eb97340af7b701aca4" candidate_uuid: "820a659649bf412b99b65c5f2629e576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477dbaca260f4a329431eeec15f4d8ac"
I20260928 22:54:08.375413 22260 leader_election.cc:302] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 820a659649bf412b99b65c5f2629e576, bc4b59c4eea843f4a1d6fb1d33640071; no voters:
I20260928 22:54:08.375447 22186 raft_consensus.cc:3037] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:08.375507 22404 raft_consensus.cc:2781] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:54:08.375646 22404 raft_consensus.cc:686] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [term 1 LEADER]: Becoming Leader. State: Replica: 820a659649bf412b99b65c5f2629e576, State: Running, Role: LEADER
I20260928 22:54:08.375720 22404 consensus_queue.cc:237] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.376147 22186 raft_consensus.cc:2454] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 820a659649bf412b99b65c5f2629e576 in term 1.
I20260928 22:54:08.376498 21889 catalog_manager.cc:5495] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 820a659649bf412b99b65c5f2629e576 (127.14.14.131). New cstate: current_term: 1 leader_uuid: "820a659649bf412b99b65c5f2629e576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } health_report { overall_health: HEALTHY } } }
Sep 28 22:54:08 dist-test-slave-kzfr krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636047, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.190@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
I20260928 22:54:08.425691 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.418491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.14.14.190:38855)
0928 22:54:08.418580 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.418583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.418692 (+ 109us) server_negotiation.cc:407] Connection header received
0928 22:54:08.418795 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.418796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.418806 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.418875 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:08.419338 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.419849 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.420553 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.420740 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.420836 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.422970 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:08.422972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:08.422972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:08.422978 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:08.424445 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:08.424849 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:08.424850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:08.424851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:08.424904 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:08.425129 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:08.425130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:08.425130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:08.425242 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:08.425315 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.425473 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.425556 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 22:54:08.425763 22437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.418420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38855 (local address 127.0.0.1:54520)
0928 22:54:08.418618 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.418627 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.418712 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.418879 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.418880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.419206 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:08.419254 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.419256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.419933 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.419934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.420448 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.420457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.420838 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.420839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.420892 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.421409 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:08.421414 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:08.422858 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:08.424533 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:08.424541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:08.424544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:08.424771 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:08.424992 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:08.424993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:08.424993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:08.425055 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:08.425306 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:08.425314 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:08.425357 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.425471 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.425638 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":43,"thread_start_us":113,"threads_started":1}
I20260928 22:54:08.427606 21889 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:54520:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 22:54:08.427788 21889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:54:08.429203 22026 tablet_service.cc:1467] Processing CreateTablet for tablet 18a852508d2a414d92e5af0f25969c5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fdcc6aa7acb9437eb5790774e71c1f35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:54:08.429298 22306 tablet_service.cc:1467] Processing CreateTablet for tablet 18a852508d2a414d92e5af0f25969c5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fdcc6aa7acb9437eb5790774e71c1f35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:54:08.429324 22166 tablet_service.cc:1467] Processing CreateTablet for tablet 18a852508d2a414d92e5af0f25969c5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fdcc6aa7acb9437eb5790774e71c1f35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:54:08.429379 22026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a852508d2a414d92e5af0f25969c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:08.429430 22306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a852508d2a414d92e5af0f25969c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:08.429488 22166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a852508d2a414d92e5af0f25969c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:08.430882 22406 tablet_bootstrap.cc:492] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac: Bootstrap starting.
I20260928 22:54:08.430976 22401 tablet_bootstrap.cc:492] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071: Bootstrap starting.
I20260928 22:54:08.430975 22400 tablet_bootstrap.cc:492] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576: Bootstrap starting.
I20260928 22:54:08.431245 22406 tablet_bootstrap.cc:654] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:08.431409 22400 tablet_bootstrap.cc:654] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:08.431411 22401 tablet_bootstrap.cc:654] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:08.431857 22406 tablet_bootstrap.cc:492] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac: No bootstrap required, opened a new log
I20260928 22:54:08.431910 22406 ts_tablet_manager.cc:1397] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:08.432050 22406 raft_consensus.cc:348] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.432122 22406 raft_consensus.cc:374] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:08.432118 22400 tablet_bootstrap.cc:492] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576: No bootstrap required, opened a new log
I20260928 22:54:08.432138 22406 raft_consensus.cc:729] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477dbaca260f4a329431eeec15f4d8ac, State: Initialized, Role: FOLLOWER
I20260928 22:54:08.432130 22401 tablet_bootstrap.cc:492] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071: No bootstrap required, opened a new log
I20260928 22:54:08.432170 22401 ts_tablet_manager.cc:1397] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:08.432166 22400 ts_tablet_manager.cc:1397] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:08.432204 22406 consensus_queue.cc:260] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.432308 22406 ts_tablet_manager.cc:1428] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:08.432312 22401 raft_consensus.cc:348] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.432379 22401 raft_consensus.cc:374] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:08.432397 22401 raft_consensus.cc:729] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4b59c4eea843f4a1d6fb1d33640071, State: Initialized, Role: FOLLOWER
I20260928 22:54:08.432402 22419 tablet_replica.cc:439] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } }
I20260928 22:54:08.432446 22401 consensus_queue.cc:260] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.432498 22419 tablet_replica.cc:442] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:54:08.432580 22401 ts_tablet_manager.cc:1428] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:08.432586 22400 raft_consensus.cc:348] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.432654 22400 raft_consensus.cc:374] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:08.432682 22400 raft_consensus.cc:729] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820a659649bf412b99b65c5f2629e576, State: Initialized, Role: FOLLOWER
I20260928 22:54:08.432647 22405 tablet_replica.cc:439] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } }
I20260928 22:54:08.432726 22400 consensus_queue.cc:260] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.432755 22405 tablet_replica.cc:442] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:54:08.432799 22400 ts_tablet_manager.cc:1428] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 22:54:08.432931 22404 tablet_replica.cc:439] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } }
I20260928 22:54:08.433041 22404 tablet_replica.cc:442] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:54:08.528553 22404 raft_consensus.cc:482] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:54:08.528646 22404 raft_consensus.cc:504] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.528820 22404 leader_election.cc:288] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc4b59c4eea843f4a1d6fb1d33640071 (127.14.14.129:44925), 477dbaca260f4a329431eeec15f4d8ac (127.14.14.130:37249)
I20260928 22:54:08.529042 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a852508d2a414d92e5af0f25969c5d" candidate_uuid: "820a659649bf412b99b65c5f2629e576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" is_pre_election: true
I20260928 22:54:08.529124 22046 raft_consensus.cc:2454] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 820a659649bf412b99b65c5f2629e576 in term 0.
I20260928 22:54:08.529141 22186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a852508d2a414d92e5af0f25969c5d" candidate_uuid: "820a659649bf412b99b65c5f2629e576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477dbaca260f4a329431eeec15f4d8ac" is_pre_election: true
I20260928 22:54:08.529258 22186 raft_consensus.cc:2454] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 820a659649bf412b99b65c5f2629e576 in term 0.
I20260928 22:54:08.529277 22260 leader_election.cc:302] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 820a659649bf412b99b65c5f2629e576, bc4b59c4eea843f4a1d6fb1d33640071; no voters:
I20260928 22:54:08.529373 22404 raft_consensus.cc:2781] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:54:08.529421 22404 raft_consensus.cc:482] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:54:08.529441 22404 raft_consensus.cc:3037] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:08.530196 22404 raft_consensus.cc:504] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.530336 22404 leader_election.cc:288] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [CANDIDATE]: Term 1 election: Requested vote from peers bc4b59c4eea843f4a1d6fb1d33640071 (127.14.14.129:44925), 477dbaca260f4a329431eeec15f4d8ac (127.14.14.130:37249)
I20260928 22:54:08.530494 22186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a852508d2a414d92e5af0f25969c5d" candidate_uuid: "820a659649bf412b99b65c5f2629e576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477dbaca260f4a329431eeec15f4d8ac"
I20260928 22:54:08.530571 22186 raft_consensus.cc:3037] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:08.530597 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a852508d2a414d92e5af0f25969c5d" candidate_uuid: "820a659649bf412b99b65c5f2629e576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc4b59c4eea843f4a1d6fb1d33640071"
I20260928 22:54:08.530699 22046 raft_consensus.cc:3037] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:08.531355 22186 raft_consensus.cc:2454] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 820a659649bf412b99b65c5f2629e576 in term 1.
I20260928 22:54:08.531387 22046 raft_consensus.cc:2454] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 820a659649bf412b99b65c5f2629e576 in term 1.
I20260928 22:54:08.531559 22258 leader_election.cc:302] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 477dbaca260f4a329431eeec15f4d8ac, 820a659649bf412b99b65c5f2629e576; no voters:
I20260928 22:54:08.531685 22404 raft_consensus.cc:2781] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:54:08.531747 22404 raft_consensus.cc:686] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [term 1 LEADER]: Becoming Leader. State: Replica: 820a659649bf412b99b65c5f2629e576, State: Running, Role: LEADER
I20260928 22:54:08.531804 22404 consensus_queue.cc:237] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } }
I20260928 22:54:08.532123 22430 tablet_replica.cc:439] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576: TxnStatusTablet state changed. Reason: New leader 820a659649bf412b99b65c5f2629e576. Latest consensus state: current_term: 1 leader_uuid: "820a659649bf412b99b65c5f2629e576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } }
I20260928 22:54:08.532222 22430 tablet_replica.cc:442] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:54:08.532414 22442 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 22:54:08.532554 22442 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 22:54:08.532689 21889 catalog_manager.cc:5495] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 820a659649bf412b99b65c5f2629e576 (127.14.14.131). New cstate: current_term: 1 leader_uuid: "820a659649bf412b99b65c5f2629e576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } health_report { overall_health: UNKNOWN } } }
I20260928 22:54:08.550726 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.547978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36158 (local address 127.14.14.131:37017)
0928 22:54:08.548024 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.548026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.548074 (+ 48us) server_negotiation.cc:407] Connection header received
0928 22:54:08.548173 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.548174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.548189 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.548240 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:54:08.548363 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.548825 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.549943 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.550158 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.550214 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.550419 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:54:08.550516 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:54:08.550570 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.550653 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.550671 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:54:08.550786 22437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.547926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37017 (local address 127.0.0.1:36158)
0928 22:54:08.548014 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.548027 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.548096 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.548253 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.548254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.548255 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:54:08.548309 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.548311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.548897 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.548898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.549860 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.549864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.550238 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.550239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.550311 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.550329 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:54:08.550585 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:54:08.550593 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.550651 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.550704 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 22:54:08.551973 22046 raft_consensus.cc:1261] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Refusing update from remote peer 820a659649bf412b99b65c5f2629e576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:08.552064 22186 raft_consensus.cc:1261] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Refusing update from remote peer 820a659649bf412b99b65c5f2629e576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:08.552237 22430 consensus_queue.cc:1035] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:08.552350 22404 consensus_queue.cc:1035] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:08.553103 22405 tablet_replica.cc:439] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071: TxnStatusTablet state changed. Reason: New leader 820a659649bf412b99b65c5f2629e576. Latest consensus state: current_term: 1 leader_uuid: "820a659649bf412b99b65c5f2629e576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } }
I20260928 22:54:08.553200 22405 tablet_replica.cc:442] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:54:08.553285 22419 tablet_replica.cc:439] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac: TxnStatusTablet state changed. Reason: New leader 820a659649bf412b99b65c5f2629e576. Latest consensus state: current_term: 1 leader_uuid: "820a659649bf412b99b65c5f2629e576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } }
I20260928 22:54:08.553367 22419 tablet_replica.cc:442] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:54:08.553716 22430 tablet_replica.cc:439] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "820a659649bf412b99b65c5f2629e576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } }
I20260928 22:54:08.553805 22430 tablet_replica.cc:442] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:54:08.553910 22405 tablet_replica.cc:439] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "820a659649bf412b99b65c5f2629e576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } }
I20260928 22:54:08.553982 22405 tablet_replica.cc:442] T 18a852508d2a414d92e5af0f25969c5d P bc4b59c4eea843f4a1d6fb1d33640071: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:54:08.554939 22430 tablet_replica.cc:439] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "820a659649bf412b99b65c5f2629e576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } }
I20260928 22:54:08.555028 22430 tablet_replica.cc:442] T 18a852508d2a414d92e5af0f25969c5d P 820a659649bf412b99b65c5f2629e576: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:54:08.555279 22419 tablet_replica.cc:439] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "820a659649bf412b99b65c5f2629e576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } }
I20260928 22:54:08.555359 22419 tablet_replica.cc:442] T 18a852508d2a414d92e5af0f25969c5d P 477dbaca260f4a329431eeec15f4d8ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:54:08.558322 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.555747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36170 (local address 127.14.14.131:37017)
0928 22:54:08.555856 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.555857 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.555863 (+ 6us) server_negotiation.cc:407] Connection header received
0928 22:54:08.555879 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.555879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.555886 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.555923 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:54:08.556103 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.556814 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.557543 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.557872 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.557944 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.558101 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:54:08.558134 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:54:08.558188 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.558235 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.558254 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 22:54:08.563745 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.560641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:42427 (local address 127.14.14.131:37017)
0928 22:54:08.560674 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.560676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.560829 (+ 153us) server_negotiation.cc:407] Connection header received
0928 22:54:08.560894 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.560894 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.560901 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.560941 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:54:08.561080 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.561488 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.562583 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.562750 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.562801 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.563515 (+ 714us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:54:08.563537 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:54:08.563575 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.563662 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.563677 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260928 22:54:08.563828 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.560559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37017 (local address 127.14.14.131:42427)
0928 22:54:08.560746 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.560756 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.560836 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.560953 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.560953 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.560955 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:54:08.561012 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.561013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.561559 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.561560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.562498 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.562501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.562825 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.562826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.563413 (+ 587us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.563427 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:54:08.563598 (+ 171us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:54:08.563599 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.563646 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.563686 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260928 22:54:08.566377 22186 raft_consensus.cc:1261] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Refusing update from remote peer 820a659649bf412b99b65c5f2629e576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:08.566447 22046 raft_consensus.cc:1261] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Refusing update from remote peer 820a659649bf412b99b65c5f2629e576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:08.566572 22404 consensus_queue.cc:1035] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:08.566653 22430 consensus_queue.cc:1035] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:08.567978 22415 mvcc.cc:205] Tried to move back new op lower bound from 7334445254926610432 to 7334445254147407872. Current Snapshot: MvccSnapshot[applied={T|T < 7334445254926610432}]
I20260928 22:54:08.578006 21889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:54:08.578114 21889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:54:08.579128 21889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088: Sending DeleteTablet for 3 replicas of tablet 0faebe57e3db43eb97340af7b701aca4
I20260928 22:54:08.579322 22166 tablet_service.cc:1514] Processing DeleteTablet for tablet 0faebe57e3db43eb97340af7b701aca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:08 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:50508
I20260928 22:54:08.579381 22026 tablet_service.cc:1514] Processing DeleteTablet for tablet 0faebe57e3db43eb97340af7b701aca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:08 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:36892
I20260928 22:54:08.579362 22165 tablet_service.cc:1514] Processing DeleteTablet for tablet 0faebe57e3db43eb97340af7b701aca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:08 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:50508
I20260928 22:54:08.579460 22425 tablet_replica.cc:330] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac: stopping tablet replica
I20260928 22:54:08.579488 22423 tablet_replica.cc:330] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071: stopping tablet replica
I20260928 22:54:08.579526 22423 raft_consensus.cc:2229] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:54:08.579512 22425 raft_consensus.cc:2229] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:54:08.579572 22425 raft_consensus.cc:2258] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:08.579598 22423 raft_consensus.cc:2258] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:08.579842 22425 ts_tablet_manager.cc:1905] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:08.580112 22306 tablet_service.cc:1514] Processing DeleteTablet for tablet 0faebe57e3db43eb97340af7b701aca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:08 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:36146
I20260928 22:54:08.580196 22424 tablet_replica.cc:330] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576: stopping tablet replica
I20260928 22:54:08.580243 22424 raft_consensus.cc:2229] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:54:08.580343 22424 raft_consensus.cc:2258] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:08.581823 22425 ts_tablet_manager.cc:1918] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:54:08.581877 22425 log.cc:1196] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/0faebe57e3db43eb97340af7b701aca4
I20260928 22:54:08.582093 22425 ts_tablet_manager.cc:1939] T 0faebe57e3db43eb97340af7b701aca4 P 477dbaca260f4a329431eeec15f4d8ac: Deleting consensus metadata
I20260928 22:54:08.582808 22424 ts_tablet_manager.cc:1905] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:08.582960 21874 catalog_manager.cc:4841] TS 477dbaca260f4a329431eeec15f4d8ac (127.14.14.130:37249): tablet 0faebe57e3db43eb97340af7b701aca4 (table test-table [id=1a20435bf1c6426195e0a82dee13d8df]) successfully deleted
W20260928 22:54:08.583099 21874 catalog_manager.cc:4808] TS 477dbaca260f4a329431eeec15f4d8ac (127.14.14.130:37249): delete failed for tablet 0faebe57e3db43eb97340af7b701aca4 because the tablet was not found. No further retry: Not found: Tablet not found: 0faebe57e3db43eb97340af7b701aca4
I20260928 22:54:08.584363 22424 ts_tablet_manager.cc:1918] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:54:08.584427 22424 log.cc:1196] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/0faebe57e3db43eb97340af7b701aca4
I20260928 22:54:08.584640 22424 ts_tablet_manager.cc:1939] T 0faebe57e3db43eb97340af7b701aca4 P 820a659649bf412b99b65c5f2629e576: Deleting consensus metadata
I20260928 22:54:08.584960 22423 ts_tablet_manager.cc:1905] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:08.585098 21874 catalog_manager.cc:4841] TS 820a659649bf412b99b65c5f2629e576 (127.14.14.131:37017): tablet 0faebe57e3db43eb97340af7b701aca4 (table test-table [id=1a20435bf1c6426195e0a82dee13d8df]) successfully deleted
I20260928 22:54:08.586376 22423 ts_tablet_manager.cc:1918] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:54:08.586450 22423 log.cc:1196] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/wal/wals/0faebe57e3db43eb97340af7b701aca4
I20260928 22:54:08.586656 22423 ts_tablet_manager.cc:1939] T 0faebe57e3db43eb97340af7b701aca4 P bc4b59c4eea843f4a1d6fb1d33640071: Deleting consensus metadata
I20260928 22:54:08.586968 21876 catalog_manager.cc:4841] TS bc4b59c4eea843f4a1d6fb1d33640071 (127.14.14.129:44925): tablet 0faebe57e3db43eb97340af7b701aca4 (table test-table [id=1a20435bf1c6426195e0a82dee13d8df]) successfully deleted
Sep 28 22:54:08 dist-test-slave-kzfr krb5kdc[21844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 22:54:08 dist-test-slave-kzfr krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:08 dist-test-slave-kzfr krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 22:54:08.613552 21938 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:08.601791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.14.14.190:38855)
0928 22:54:08.601846 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.601849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.601991 (+ 142us) server_negotiation.cc:407] Connection header received
0928 22:54:08.602078 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.602079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.602090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.602171 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:08.602806 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.603303 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.604074 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.604281 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.604368 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.613446 (+ 9078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54532: BlockingRecv error: recv got EOF from 127.0.0.1:54532 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 22:54:08 dist-test-slave-kzfr krb5kdc[21844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM
I20260928 22:54:08.622121 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.614505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54544 (local address 127.14.14.190:38855)
0928 22:54:08.614553 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.614555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.614590 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:54:08.614624 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.614625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.614637 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.614700 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:08.615213 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.615959 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.616749 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.617044 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.617127 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.619493 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:08.619495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:08.619495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:08.619501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:08.620889 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:08.621303 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:08.621304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:08.621304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:08.621355 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:08.621563 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:08.621564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:08.621564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:08.621728 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:08.621802 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.621995 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.622057 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:54:08.623912 21888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:54:08.624102 21888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:54:08.625515 22165 tablet_service.cc:1467] Processing CreateTablet for tablet 44d01271939e4f4b9a21c6adf07c62ef (DEFAULT_TABLE table=test-table [id=4cdbca0ff69b4053ab3bcb87f98a67b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:08.625543 22026 tablet_service.cc:1467] Processing CreateTablet for tablet 44d01271939e4f4b9a21c6adf07c62ef (DEFAULT_TABLE table=test-table [id=4cdbca0ff69b4053ab3bcb87f98a67b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:08.625710 22026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d01271939e4f4b9a21c6adf07c62ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:08.625712 22165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d01271939e4f4b9a21c6adf07c62ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:08.626809 22406 tablet_bootstrap.cc:492] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac: Bootstrap starting.
I20260928 22:54:08.627135 22406 tablet_bootstrap.cc:654] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:08.627167 22401 tablet_bootstrap.cc:492] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071: Bootstrap starting.
I20260928 22:54:08.627200 22306 tablet_service.cc:1467] Processing CreateTablet for tablet 44d01271939e4f4b9a21c6adf07c62ef (DEFAULT_TABLE table=test-table [id=4cdbca0ff69b4053ab3bcb87f98a67b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:08.627318 22306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d01271939e4f4b9a21c6adf07c62ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:08.627599 22401 tablet_bootstrap.cc:654] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:08.628154 22401 tablet_bootstrap.cc:492] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071: No bootstrap required, opened a new log
I20260928 22:54:08.628223 22401 ts_tablet_manager.cc:1397] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:08.628381 22401 raft_consensus.cc:348] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.628464 22401 raft_consensus.cc:374] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:08.628494 22401 raft_consensus.cc:729] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4b59c4eea843f4a1d6fb1d33640071, State: Initialized, Role: FOLLOWER
I20260928 22:54:08.628535 22401 consensus_queue.cc:260] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.628628 22401 ts_tablet_manager.cc:1428] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:08.628650 22400 tablet_bootstrap.cc:492] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576: Bootstrap starting.
I20260928 22:54:08.629007 22400 tablet_bootstrap.cc:654] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:08.629869 22400 tablet_bootstrap.cc:492] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576: No bootstrap required, opened a new log
I20260928 22:54:08.629922 22400 ts_tablet_manager.cc:1397] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:08.630106 22400 raft_consensus.cc:348] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.630182 22400 raft_consensus.cc:374] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:08.630203 22400 raft_consensus.cc:729] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820a659649bf412b99b65c5f2629e576, State: Initialized, Role: FOLLOWER
I20260928 22:54:08.630246 22400 consensus_queue.cc:260] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.630347 22400 ts_tablet_manager.cc:1428] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:08.630985 22406 tablet_bootstrap.cc:492] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac: No bootstrap required, opened a new log
I20260928 22:54:08.631047 22406 ts_tablet_manager.cc:1397] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 22:54:08.631166 22406 raft_consensus.cc:348] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.631283 22406 raft_consensus.cc:374] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:08.631307 22406 raft_consensus.cc:729] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477dbaca260f4a329431eeec15f4d8ac, State: Initialized, Role: FOLLOWER
I20260928 22:54:08.631347 22406 consensus_queue.cc:260] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.631443 22406 ts_tablet_manager.cc:1428] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:08.643004 22405 raft_consensus.cc:482] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:54:08.643121 22405 raft_consensus.cc:504] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.643427 22405 leader_election.cc:288] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 477dbaca260f4a329431eeec15f4d8ac (127.14.14.130:37249), 820a659649bf412b99b65c5f2629e576 (127.14.14.131:37017)
I20260928 22:54:08.647117 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.643689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:49793 (local address 127.14.14.130:37249)
0928 22:54:08.643733 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.643736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.643903 (+ 167us) server_negotiation.cc:407] Connection header received
0928 22:54:08.643935 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.643936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.643946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.644039 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:08.644277 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.645049 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.646290 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.646748 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.646821 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.646830 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.647005 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.647047 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:54:08.647449 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.643647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:37249 (local address 127.14.14.129:49793)
0928 22:54:08.643777 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.643788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.643877 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.644092 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.644093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.644099 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:08.644185 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.644188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.645148 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.645149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.646189 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.646192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.646845 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.646846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.646920 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.646925 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.646992 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.647043 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":23,"thread_start_us":87,"threads_started":1}
I20260928 22:54:08.647588 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.643997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:38075 (local address 127.14.14.131:37017)
0928 22:54:08.644059 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.644061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.644091 (+ 30us) server_negotiation.cc:407] Connection header received
0928 22:54:08.644159 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.644160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.644169 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.644222 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:08.644401 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.645090 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.646609 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.647131 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.647198 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.647206 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.647472 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.647505 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:54:08.647730 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.643849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37017 (local address 127.14.14.129:38075)
0928 22:54:08.644018 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.644028 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.644105 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.644242 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.644243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.644245 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:08.644322 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.644324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.645206 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.645207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.646511 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.646514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.647335 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.647336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.647405 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.647408 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.647462 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.647498 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":78,"thread_start_us":90,"threads_started":1}
I20260928 22:54:08.647778 22186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d01271939e4f4b9a21c6adf07c62ef" candidate_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477dbaca260f4a329431eeec15f4d8ac" is_pre_election: true
I20260928 22:54:08.647884 22186 raft_consensus.cc:2454] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc4b59c4eea843f4a1d6fb1d33640071 in term 0.
I20260928 22:54:08.648008 22326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d01271939e4f4b9a21c6adf07c62ef" candidate_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820a659649bf412b99b65c5f2629e576" is_pre_election: true
I20260928 22:54:08.648090 22326 raft_consensus.cc:2454] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc4b59c4eea843f4a1d6fb1d33640071 in term 0.
I20260928 22:54:08.648061 21978 leader_election.cc:302] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 477dbaca260f4a329431eeec15f4d8ac, bc4b59c4eea843f4a1d6fb1d33640071; no voters:
I20260928 22:54:08.648188 22405 raft_consensus.cc:2781] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:54:08.648233 22405 raft_consensus.cc:482] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:54:08.648262 22405 raft_consensus.cc:3037] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:08.649199 22405 raft_consensus.cc:504] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.649327 22405 leader_election.cc:288] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [CANDIDATE]: Term 1 election: Requested vote from peers 477dbaca260f4a329431eeec15f4d8ac (127.14.14.130:37249), 820a659649bf412b99b65c5f2629e576 (127.14.14.131:37017)
I20260928 22:54:08.649547 22326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d01271939e4f4b9a21c6adf07c62ef" candidate_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820a659649bf412b99b65c5f2629e576"
I20260928 22:54:08.649616 22186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d01271939e4f4b9a21c6adf07c62ef" candidate_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477dbaca260f4a329431eeec15f4d8ac"
I20260928 22:54:08.649655 22326 raft_consensus.cc:3037] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:08.649699 22186 raft_consensus.cc:3037] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:08.650286 22326 raft_consensus.cc:2454] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc4b59c4eea843f4a1d6fb1d33640071 in term 1.
I20260928 22:54:08.650473 21978 leader_election.cc:302] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 820a659649bf412b99b65c5f2629e576, bc4b59c4eea843f4a1d6fb1d33640071; no voters:
I20260928 22:54:08.650549 22186 raft_consensus.cc:2454] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc4b59c4eea843f4a1d6fb1d33640071 in term 1.
I20260928 22:54:08.650729 22405 raft_consensus.cc:2781] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:54:08.650908 22405 raft_consensus.cc:686] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 LEADER]: Becoming Leader. State: Replica: bc4b59c4eea843f4a1d6fb1d33640071, State: Running, Role: LEADER
I20260928 22:54:08.650986 22405 consensus_queue.cc:237] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.651669 21888 catalog_manager.cc:5495] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc4b59c4eea843f4a1d6fb1d33640071 (127.14.14.129). New cstate: current_term: 1 leader_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } health_report { overall_health: UNKNOWN } } }
I20260928 22:54:08.661540 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.658694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.14.14.129:44925)
0928 22:54:08.658762 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.658764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.658773 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:54:08.658824 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.658825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.658849 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.658940 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:54:08.659108 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.659842 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.660631 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.660945 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.661021 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.661186 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:54:08.661324 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:54:08.661405 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.661431 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.661454 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:54:08.662637 22186 raft_consensus.cc:1261] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Refusing update from remote peer bc4b59c4eea843f4a1d6fb1d33640071: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:08.662647 22326 raft_consensus.cc:1261] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Refusing update from remote peer bc4b59c4eea843f4a1d6fb1d33640071: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:08.662966 22471 consensus_queue.cc:1035] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:08.663115 22405 consensus_queue.cc:1035] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:08.667145 21888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:54:08.667245 21888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:54:08.667907 21888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088: Sending DeleteTablet for 3 replicas of tablet 44d01271939e4f4b9a21c6adf07c62ef
I20260928 22:54:08.668293 22026 tablet_service.cc:1514] Processing DeleteTablet for tablet 44d01271939e4f4b9a21c6adf07c62ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:08 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:36892
I20260928 22:54:08.668394 22423 tablet_replica.cc:330] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071: stopping tablet replica
I20260928 22:54:08.668445 22423 raft_consensus.cc:2229] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:54:08.668524 22423 raft_consensus.cc:2258] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:08.668748 22165 tablet_service.cc:1514] Processing DeleteTablet for tablet 44d01271939e4f4b9a21c6adf07c62ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:08 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:50508
I20260928 22:54:08.668802 22306 tablet_service.cc:1514] Processing DeleteTablet for tablet 44d01271939e4f4b9a21c6adf07c62ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:08 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:36146
I20260928 22:54:08.668852 22425 tablet_replica.cc:330] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac: stopping tablet replica
I20260928 22:54:08.668885 22424 tablet_replica.cc:330] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576: stopping tablet replica
I20260928 22:54:08.668919 22425 raft_consensus.cc:2229] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:54:08.668933 22424 raft_consensus.cc:2229] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:54:08.668970 22424 raft_consensus.cc:2258] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:08.668974 22425 raft_consensus.cc:2258] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:08.669315 22425 ts_tablet_manager.cc:1905] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:08.669538 22424 ts_tablet_manager.cc:1905] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:08.671231 22423 ts_tablet_manager.cc:1905] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:08.671140 21888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:54:08.671319 21888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:54:08.672484 22026 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc15ec28f5b45c79d2d11da0d5fd386 (DEFAULT_TABLE table=test-table [id=0eaa295e8a4e4565aa4001b826073b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:08.672617 22423 ts_tablet_manager.cc:1918] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:54:08.672688 22423 log.cc:1196] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-0/wal/wals/44d01271939e4f4b9a21c6adf07c62ef
I20260928 22:54:08.672845 22165 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc15ec28f5b45c79d2d11da0d5fd386 (DEFAULT_TABLE table=test-table [id=0eaa295e8a4e4565aa4001b826073b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:08.672902 22423 ts_tablet_manager.cc:1939] T 44d01271939e4f4b9a21c6adf07c62ef P bc4b59c4eea843f4a1d6fb1d33640071: Deleting consensus metadata
I20260928 22:54:08.673085 22306 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc15ec28f5b45c79d2d11da0d5fd386 (DEFAULT_TABLE table=test-table [id=0eaa295e8a4e4565aa4001b826073b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:08.673623 22424 ts_tablet_manager.cc:1918] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:54:08.673672 22424 log.cc:1196] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/44d01271939e4f4b9a21c6adf07c62ef
I20260928 22:54:08.673892 22424 ts_tablet_manager.cc:1939] T 44d01271939e4f4b9a21c6adf07c62ef P 820a659649bf412b99b65c5f2629e576: Deleting consensus metadata
I20260928 22:54:08.672616 22026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc15ec28f5b45c79d2d11da0d5fd386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:08.674027 22306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc15ec28f5b45c79d2d11da0d5fd386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:08.675043 22401 tablet_bootstrap.cc:492] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071: Bootstrap starting.
I20260928 22:54:08.675287 22400 tablet_bootstrap.cc:492] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576: Bootstrap starting.
I20260928 22:54:08.675436 22401 tablet_bootstrap.cc:654] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:08.675665 22400 tablet_bootstrap.cc:654] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:08.676246 22425 ts_tablet_manager.cc:1918] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:54:08.676311 22425 log.cc:1196] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/44d01271939e4f4b9a21c6adf07c62ef
I20260928 22:54:08.676527 22425 ts_tablet_manager.cc:1939] T 44d01271939e4f4b9a21c6adf07c62ef P 477dbaca260f4a329431eeec15f4d8ac: Deleting consensus metadata
I20260928 22:54:08.676682 22165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc15ec28f5b45c79d2d11da0d5fd386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:08.677799 21874 catalog_manager.cc:4841] TS 477dbaca260f4a329431eeec15f4d8ac (127.14.14.130:37249): tablet 44d01271939e4f4b9a21c6adf07c62ef (table test-table [id=4cdbca0ff69b4053ab3bcb87f98a67b2]) successfully deleted
I20260928 22:54:08.678005 22406 tablet_bootstrap.cc:492] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac: Bootstrap starting.
I20260928 22:54:08.678162 21876 catalog_manager.cc:4841] TS bc4b59c4eea843f4a1d6fb1d33640071 (127.14.14.129:44925): tablet 44d01271939e4f4b9a21c6adf07c62ef (table test-table [id=4cdbca0ff69b4053ab3bcb87f98a67b2]) successfully deleted
I20260928 22:54:08.678305 22406 tablet_bootstrap.cc:654] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:08.678670 21874 catalog_manager.cc:4841] TS 820a659649bf412b99b65c5f2629e576 (127.14.14.131:37017): tablet 44d01271939e4f4b9a21c6adf07c62ef (table test-table [id=4cdbca0ff69b4053ab3bcb87f98a67b2]) successfully deleted
I20260928 22:54:08.678900 22400 tablet_bootstrap.cc:492] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576: No bootstrap required, opened a new log
I20260928 22:54:08.678948 22400 ts_tablet_manager.cc:1397] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 22:54:08.679015 22401 tablet_bootstrap.cc:492] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071: No bootstrap required, opened a new log
I20260928 22:54:08.679059 22401 ts_tablet_manager.cc:1397] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 22:54:08.679100 22400 raft_consensus.cc:348] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.679170 22400 raft_consensus.cc:374] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:08.679191 22400 raft_consensus.cc:729] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820a659649bf412b99b65c5f2629e576, State: Initialized, Role: FOLLOWER
I20260928 22:54:08.679194 22401 raft_consensus.cc:348] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.679232 22400 consensus_queue.cc:260] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.679261 22401 raft_consensus.cc:374] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:08.679282 22401 raft_consensus.cc:729] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4b59c4eea843f4a1d6fb1d33640071, State: Initialized, Role: FOLLOWER
I20260928 22:54:08.679312 22400 ts_tablet_manager.cc:1428] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:08.679320 22401 consensus_queue.cc:260] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.679433 22401 ts_tablet_manager.cc:1428] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:08.680145 22406 tablet_bootstrap.cc:492] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac: No bootstrap required, opened a new log
I20260928 22:54:08.680197 22406 ts_tablet_manager.cc:1397] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:54:08.680289 22406 raft_consensus.cc:348] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.680349 22406 raft_consensus.cc:374] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:08.680368 22406 raft_consensus.cc:729] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477dbaca260f4a329431eeec15f4d8ac, State: Initialized, Role: FOLLOWER
I20260928 22:54:08.680408 22406 consensus_queue.cc:260] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.680493 22406 ts_tablet_manager.cc:1428] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:08.716075 22405 raft_consensus.cc:482] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:54:08.716181 22405 raft_consensus.cc:504] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.716334 22405 leader_election.cc:288] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 477dbaca260f4a329431eeec15f4d8ac (127.14.14.130:37249), 820a659649bf412b99b65c5f2629e576 (127.14.14.131:37017)
I20260928 22:54:08.716625 22326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc15ec28f5b45c79d2d11da0d5fd386" candidate_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820a659649bf412b99b65c5f2629e576" is_pre_election: true
I20260928 22:54:08.716598 22186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc15ec28f5b45c79d2d11da0d5fd386" candidate_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477dbaca260f4a329431eeec15f4d8ac" is_pre_election: true
I20260928 22:54:08.716720 22326 raft_consensus.cc:2454] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc4b59c4eea843f4a1d6fb1d33640071 in term 0.
I20260928 22:54:08.716720 22186 raft_consensus.cc:2454] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc4b59c4eea843f4a1d6fb1d33640071 in term 0.
I20260928 22:54:08.716892 21978 leader_election.cc:302] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 477dbaca260f4a329431eeec15f4d8ac, bc4b59c4eea843f4a1d6fb1d33640071; no voters:
I20260928 22:54:08.717002 22405 raft_consensus.cc:2781] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:54:08.717051 22405 raft_consensus.cc:482] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:54:08.717069 22405 raft_consensus.cc:3037] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:08.717989 22405 raft_consensus.cc:504] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.718124 22405 leader_election.cc:288] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [CANDIDATE]: Term 1 election: Requested vote from peers 477dbaca260f4a329431eeec15f4d8ac (127.14.14.130:37249), 820a659649bf412b99b65c5f2629e576 (127.14.14.131:37017)
I20260928 22:54:08.718302 22186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc15ec28f5b45c79d2d11da0d5fd386" candidate_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477dbaca260f4a329431eeec15f4d8ac"
I20260928 22:54:08.718317 22326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc15ec28f5b45c79d2d11da0d5fd386" candidate_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820a659649bf412b99b65c5f2629e576"
I20260928 22:54:08.718371 22186 raft_consensus.cc:3037] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:08.718376 22326 raft_consensus.cc:3037] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:08.719139 22326 raft_consensus.cc:2454] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc4b59c4eea843f4a1d6fb1d33640071 in term 1.
I20260928 22:54:08.719151 22186 raft_consensus.cc:2454] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc4b59c4eea843f4a1d6fb1d33640071 in term 1.
I20260928 22:54:08.719292 21978 leader_election.cc:302] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 477dbaca260f4a329431eeec15f4d8ac, bc4b59c4eea843f4a1d6fb1d33640071; no voters:
I20260928 22:54:08.719389 22405 raft_consensus.cc:2781] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:54:08.719435 22405 raft_consensus.cc:686] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 LEADER]: Becoming Leader. State: Replica: bc4b59c4eea843f4a1d6fb1d33640071, State: Running, Role: LEADER
I20260928 22:54:08.719481 22405 consensus_queue.cc:237] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } }
I20260928 22:54:08.720052 21887 catalog_manager.cc:5495] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc4b59c4eea843f4a1d6fb1d33640071 (127.14.14.129). New cstate: current_term: 1 leader_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4b59c4eea843f4a1d6fb1d33640071" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 } health_report { overall_health: UNKNOWN } } }
I20260928 22:54:08.736704 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.731183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:54529 (local address 127.14.14.131:37017)
0928 22:54:08.731229 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.731231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.733806 (+ 2575us) server_negotiation.cc:407] Connection header received
0928 22:54:08.733862 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.733863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.733875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.733939 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:54:08.734069 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.734820 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.735882 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.736168 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.736238 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.736424 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:54:08.736460 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:54:08.736514 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.736600 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.736625 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:54:08.736716 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.728448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37017 (local address 127.14.14.129:54529)
0928 22:54:08.733715 (+ 5267us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.733727 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.733801 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.733939 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.733940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.733942 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:54:08.734001 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.734003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.734916 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.734918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.735784 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.735787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.736256 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.736257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.736316 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.736331 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:54:08.736531 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:54:08.736533 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.736584 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.736636 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5219,"thread_start_us":79,"threads_started":1}
I20260928 22:54:08.739792 22186 raft_consensus.cc:1261] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Refusing update from remote peer bc4b59c4eea843f4a1d6fb1d33640071: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:08.739809 22326 raft_consensus.cc:1261] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Refusing update from remote peer bc4b59c4eea843f4a1d6fb1d33640071: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:08.740025 22471 consensus_queue.cc:1035] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820a659649bf412b99b65c5f2629e576" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:08.740137 22471 consensus_queue.cc:1035] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "477dbaca260f4a329431eeec15f4d8ac" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:08.749258 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.746274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:44925 (local address 127.14.14.131:46939)
0928 22:54:08.746320 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.746331 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:08.746365 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:08.746527 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:08.746529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:08.746531 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:54:08.746573 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.746574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.747463 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.747464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.748327 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:08.748329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.748797 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.748798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:08.748857 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.748865 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:54:08.749079 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:54:08.749081 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:54:08.749128 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 22:54:08.749175 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 22:54:08.749433 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.746342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:46939 (local address 127.14.14.129:44925)
0928 22:54:08.746380 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:08.746382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:08.746390 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:54:08.746414 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:08.746415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:08.746424 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:08.746479 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:54:08.746641 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.747373 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.748425 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:08.748705 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:08.748781 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:08.748970 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:54:08.749007 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:54:08.749065 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:08.749349 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 22:54:08.749371 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:54:08.756359 21887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:54:08.756465 21887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:54:08.757084 21887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1fdfd4743a84e26abd80f95a1ad1088: Sending DeleteTablet for 3 replicas of tablet 6bc15ec28f5b45c79d2d11da0d5fd386
I20260928 22:54:08.757334 22026 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bc15ec28f5b45c79d2d11da0d5fd386 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:08 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:36892
I20260928 22:54:08.757417 22423 tablet_replica.cc:330] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071: stopping tablet replica
I20260928 22:54:08.757460 22423 raft_consensus.cc:2229] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:54:08.757514 22423 raft_consensus.cc:2258] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:08.757958 22423 ts_tablet_manager.cc:1905] T 6bc15ec28f5b45c79d2d11da0d5fd386 P bc4b59c4eea843f4a1d6fb1d33640071: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:08.758025 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 21961
I20260928 22:54:08.759874 22165 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bc15ec28f5b45c79d2d11da0d5fd386 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:08 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:50508
I20260928 22:54:08.759979 22425 tablet_replica.cc:330] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac: stopping tablet replica
I20260928 22:54:08.759991 22306 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bc15ec28f5b45c79d2d11da0d5fd386 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:08 UTC) from {username='oryx', principal='oryx/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:36146
I20260928 22:54:08.760025 22425 raft_consensus.cc:2229] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:54:08.760071 22425 raft_consensus.cc:2258] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:08.760084 22424 tablet_replica.cc:330] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576: stopping tablet replica
I20260928 22:54:08.760131 22424 raft_consensus.cc:2229] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:54:08.760165 22424 raft_consensus.cc:2258] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:08.760399 22425 ts_tablet_manager.cc:1905] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:08.760663 22424 ts_tablet_manager.cc:1905] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:08.762005 22425 ts_tablet_manager.cc:1918] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:54:08.762069 22425 log.cc:1196] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/6bc15ec28f5b45c79d2d11da0d5fd386
I20260928 22:54:08.762293 22425 ts_tablet_manager.cc:1939] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 477dbaca260f4a329431eeec15f4d8ac: Deleting consensus metadata
I20260928 22:54:08.762481 22424 ts_tablet_manager.cc:1918] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:54:08.762532 22424 log.cc:1196] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/6bc15ec28f5b45c79d2d11da0d5fd386
I20260928 22:54:08.762633 21874 catalog_manager.cc:4841] TS 477dbaca260f4a329431eeec15f4d8ac (127.14.14.130:37249): tablet 6bc15ec28f5b45c79d2d11da0d5fd386 (table test-table [id=0eaa295e8a4e4565aa4001b826073b02]) successfully deleted
I20260928 22:54:08.762730 22424 ts_tablet_manager.cc:1939] T 6bc15ec28f5b45c79d2d11da0d5fd386 P 820a659649bf412b99b65c5f2629e576: Deleting consensus metadata
I20260928 22:54:08.763123 21874 catalog_manager.cc:4841] TS 820a659649bf412b99b65c5f2629e576 (127.14.14.131:37017): tablet 6bc15ec28f5b45c79d2d11da0d5fd386 (table test-table [id=0eaa295e8a4e4565aa4001b826073b02]) successfully deleted
W20260928 22:54:08.764642 21876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:44925 (error 108)
I20260928 22:54:08.764880 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 22101
I20260928 22:54:08.765599 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:08.764966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:44925 (local address 127.0.0.1:36910)
0928 22:54:08.765487 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:08.765531 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:44925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":41,"threads_started":1}
W20260928 22:54:08.765688 21876 catalog_manager.cc:4584] TS bc4b59c4eea843f4a1d6fb1d33640071 (127.14.14.129:44925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6bc15ec28f5b45c79d2d11da0d5fd386: Network error: Client connection negotiation failed: client connection to 127.14.14.129:44925: connect: Connection refused (error 111)
I20260928 22:54:08.770977 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 22241
I20260928 22:54:08.776723 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 21857
2026-09-28T22:54:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2600 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:54:08 dist-test-slave-kzfr krb5kdc[22490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:54:08 dist-test-slave-kzfr krb5kdc[22490](info): set up 2 sockets
Sep 28 22:54:08 dist-test-slave-kzfr krb5kdc[22490](info): commencing operation
krb5kdc: starting...
W20260928 22:54:09.808435 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:54:09 dist-test-slave-kzfr krb5kdc[22490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:54:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:54:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:54:09.928318 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:40001
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:37371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:40001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:10.008318 22503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:10.008522 22503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:10.008543 22503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:10.009718 22503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:54:10.009758 22503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:10.009773 22503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:10.009785 22503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:54:10.009842 22503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:54:10.011291 22503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:37371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:40001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:40001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:10.011541 22503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:10.011775 22503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:10.016671 22511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:10.016681 22508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:10.016691 22509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:10.016934 22503 server_base.cc:1034] running on GCE node
I20260928 22:54:10.017150 22503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:10.017400 22503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:10.018592 22503 hybrid_clock.cc:648] HybridClock initialized: now 1790636050018576 us; error 26 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T22:54:10Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1248 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:54:10 dist-test-slave-kzfr krb5kdc[22516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:54:10 dist-test-slave-kzfr krb5kdc[22516](info): set up 2 sockets
Sep 28 22:54:10 dist-test-slave-kzfr krb5kdc[22516](info): commencing operation
krb5kdc: starting...
W20260928 22:54:11.074522 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:54:11 dist-test-slave-kzfr krb5kdc[22516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:54:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:54:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:54:11.192663 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:35131
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:35131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:11.272226 22529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:11.272429 22529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:11.272451 22529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:11.273703 22529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:54:11.273746 22529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:11.273761 22529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:11.273792 22529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:54:11.273821 22529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:54:11.275269 22529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:35131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:35131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:11.275487 22529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:11.275696 22529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:11.280633 22535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:11.280675 22534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:11.280646 22537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:11.280774 22529 server_base.cc:1034] running on GCE node
I20260928 22:54:11.281136 22529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:11.281422 22529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:11.282608 22529 hybrid_clock.cc:648] HybridClock initialized: now 1790636051282591 us; error 34 us; skew 500 ppm
Sep 28 22:54:11 dist-test-slave-kzfr krb5kdc[22516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636051, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:11.285102 22529 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:54:11.285573 22529 webserver.cc:460] Webserver started at http://127.14.14.190:42457/ using document root <none> and password file <none>
I20260928 22:54:11.285856 22529 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:11.285923 22529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:11.286074 22529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:11.287132 22529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "67555a2cc798471eb687be38a1f4f206"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "df9492a170c834500e2694ae56b88112"
server_key_iv: "f8adada12bb1003a3c0b53ed3fd4f1c6"
server_key_version: "encryptionkey@0"
I20260928 22:54:11.287506 22529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "67555a2cc798471eb687be38a1f4f206"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "df9492a170c834500e2694ae56b88112"
server_key_iv: "f8adada12bb1003a3c0b53ed3fd4f1c6"
server_key_version: "encryptionkey@0"
I20260928 22:54:11.288875 22529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:54:11.289665 22544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:11.289840 22529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:11.289942 22529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "67555a2cc798471eb687be38a1f4f206"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "df9492a170c834500e2694ae56b88112"
server_key_iv: "f8adada12bb1003a3c0b53ed3fd4f1c6"
server_key_version: "encryptionkey@0"
I20260928 22:54:11.290022 22529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:11.303732 22529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:11.305676 22529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:11.305853 22529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:11.309911 22529 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:35131
I20260928 22:54:11.309947 22596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:35131 every 8 connection(s)
I20260928 22:54:11.310326 22529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:54:11.311156 22597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:11.313320 22597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206: Bootstrap starting.
I20260928 22:54:11.313874 22597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:11.314103 22597 log.cc:824] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:11.314735 22597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206: No bootstrap required, opened a new log
I20260928 22:54:11.316200 22597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67555a2cc798471eb687be38a1f4f206" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35131 } }
I20260928 22:54:11.316331 22597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:11.316354 22597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67555a2cc798471eb687be38a1f4f206, State: Initialized, Role: FOLLOWER
I20260928 22:54:11.316498 22597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67555a2cc798471eb687be38a1f4f206" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35131 } }
I20260928 22:54:11.316584 22597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:54:11.316607 22597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:54:11.316635 22597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:11.317142 22597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67555a2cc798471eb687be38a1f4f206" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35131 } }
I20260928 22:54:11.317245 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 22529
I20260928 22:54:11.317263 22597 leader_election.cc:302] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67555a2cc798471eb687be38a1f4f206; no voters:
I20260928 22:54:11.317366 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:54:11.317425 22597 leader_election.cc:288] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:54:11.317533 22600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:54:11.317643 14394 external_mini_cluster.cc:1428] Setting key f5beb88b5ae21e7a240cbe847c92ab38
I20260928 22:54:11.317701 22597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:54:11.317723 22600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 1 LEADER]: Becoming Leader. State: Replica: 67555a2cc798471eb687be38a1f4f206, State: Running, Role: LEADER
I20260928 22:54:11.317807 22600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67555a2cc798471eb687be38a1f4f206" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35131 } }
I20260928 22:54:11.318099 22600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67555a2cc798471eb687be38a1f4f206. Latest consensus state: current_term: 1 leader_uuid: "67555a2cc798471eb687be38a1f4f206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67555a2cc798471eb687be38a1f4f206" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35131 } } }
I20260928 22:54:11.318143 22601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67555a2cc798471eb687be38a1f4f206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67555a2cc798471eb687be38a1f4f206" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35131 } } }
I20260928 22:54:11.318182 22600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:11.318197 22601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:11.318389 22606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:54:11.318886 22606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:54:11.320775 22606 catalog_manager.cc:1347] Generated new cluster ID: dc2bd9b135dd4f01a0a3e3b002209b5a
I20260928 22:54:11.320827 22606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:54:11 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:54:11.327987 22606 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:54:11.328717 22606 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:54:11.330168 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:11.318185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51476 (local address 127.14.14.190:35131)
0928 22:54:11.318337 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:11.318346 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:11.318706 (+ 360us) server_negotiation.cc:407] Connection header received
0928 22:54:11.318831 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:11.318838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:11.318905 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:11.319056 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:11.320023 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:11.321734 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:11.322770 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:11.323390 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:11.323477 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:11.326214 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:11.326219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:11.326219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:11.326230 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:11.328239 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:11.328626 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:11.328629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:11.328629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:11.328724 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:11.328926 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:11.328926 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:11.328927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:11.329176 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:11.329304 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:11.329789 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 22:54:11.329925 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260928 22:54:11.334161 22606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206: Generated new TSK 0
I20260928 22:54:11.334326 22606 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:54:11.364852 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:35131
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:11.450338 22623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:11.450570 22623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:11.450598 22623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:11.451910 22623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:11.451967 22623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:11.451997 22623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:54:11.453641 22623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:35131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:11.453922 22623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:11.454195 22623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:11.459683 22628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:11.459692 22631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:11.459687 22629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:11.459988 22623 server_base.cc:1034] running on GCE node
I20260928 22:54:11.460256 22623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:11.460476 22623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:11.461666 22623 hybrid_clock.cc:648] HybridClock initialized: now 1790636051461647 us; error 26 us; skew 500 ppm
Sep 28 22:54:11 dist-test-slave-kzfr krb5kdc[22516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636051, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:11.464406 22623 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:54:11.464915 22623 webserver.cc:460] Webserver started at http://127.14.14.129:38353/ using document root <none> and password file <none>
I20260928 22:54:11.465137 22623 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:11.465186 22623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:11.465283 22623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:11.466564 22623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "79fc4b0ee59642f8b765ed4afcee4ecd"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "4e2f5a8433a1809f0ea628b62eb003e1"
server_key_iv: "5b7f062bfa30fea293416bd467f77905"
server_key_version: "encryptionkey@0"
I20260928 22:54:11.466892 22623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "79fc4b0ee59642f8b765ed4afcee4ecd"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "4e2f5a8433a1809f0ea628b62eb003e1"
server_key_iv: "5b7f062bfa30fea293416bd467f77905"
server_key_version: "encryptionkey@0"
I20260928 22:54:11.468267 22623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:54:11.469111 22638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:11.469354 22623 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:11.469467 22623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "79fc4b0ee59642f8b765ed4afcee4ecd"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "4e2f5a8433a1809f0ea628b62eb003e1"
server_key_iv: "5b7f062bfa30fea293416bd467f77905"
server_key_version: "encryptionkey@0"
I20260928 22:54:11.469542 22623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:11.480152 22623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:11.482435 22623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:11.482547 22623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:11.482749 22623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:11.483078 22623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:11.483112 22623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:11.483161 22623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:11.483188 22623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:11.488950 22623 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:43771
I20260928 22:54:11.489132 22751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:43771 every 8 connection(s)
I20260928 22:54:11.489400 22623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:54:11.489809 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 22623
I20260928 22:54:11.489924 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:54:11.490142 14394 external_mini_cluster.cc:1428] Setting key 640570ae198baab5248c029c049a29cb
Sep 28 22:54:11 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636051, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
I20260928 22:54:11.500015 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:11.490291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:38819 (local address 127.14.14.190:35131)
0928 22:54:11.490357 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:11.490360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:11.490653 (+ 293us) server_negotiation.cc:407] Connection header received
0928 22:54:11.491231 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:11.491232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:11.491243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:11.491293 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:11.492321 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:11.492946 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:11.494103 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:11.494323 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:11.494409 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:11.496844 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:11.496845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:11.496846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:11.496854 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:11.498534 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:11.498971 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:11.498973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:11.498973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:11.499036 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:11.499303 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:11.499304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:11.499305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:11.499437 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:11.499521 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:11.499812 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 22:54:11.499920 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:54:11.500204 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:11.490293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:38819)
0928 22:54:11.490515 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:11.490565 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:11.491111 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:11.491366 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:11.491370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:11.492021 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:11.492219 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:11.492224 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:11.493052 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:11.493053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:11.494004 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:11.494006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:11.494414 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:11.494415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:11.494540 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:11.495107 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:11.495116 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:11.496739 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:11.498629 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:11.498637 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:11.498640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:11.498876 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:11.499109 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:11.499110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:11.499110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:11.499233 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:11.499511 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:11.499517 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:11.499669 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 22:54:11.499778 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 22:54:11.499950 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260928 22:54:11.500830 22752 heartbeater.cc:347] Connected to a master server at 127.14.14.190:35131
I20260928 22:54:11.500972 22752 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
I20260928 22:54:11.501212 22752 heartbeater.cc:511] Master 127.14.14.190:35131 requested a full tablet report, sending...
I20260928 22:54:11.501869 22561 ts_manager.cc:194] Registered new tserver with Master: 79fc4b0ee59642f8b765ed4afcee4ecd (127.14.14.129:43771)
I20260928 22:54:11.502542 22561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:38819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:54:11.517879 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:35131
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:11.599403 22757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:11.599599 22757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:11.599623 22757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:11.600692 22757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:11.600728 22757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:11.600754 22757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:54:11.602149 22757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:35131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:11.602319 22757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:11.602571 22757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:11.607538 22763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:11.607563 22762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:11.607604 22757 server_base.cc:1034] running on GCE node
W20260928 22:54:11.607568 22765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:11.608001 22757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:11.608243 22757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:11.609403 22757 hybrid_clock.cc:648] HybridClock initialized: now 1790636051609399 us; error 39 us; skew 500 ppm
Sep 28 22:54:11 dist-test-slave-kzfr krb5kdc[22516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636051, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:11.611934 22757 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:54:11.612484 22757 webserver.cc:460] Webserver started at http://127.14.14.130:32905/ using document root <none> and password file <none>
I20260928 22:54:11.612751 22757 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:11.612798 22757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:11.612888 22757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:11.613958 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "36c03d23300f439da64ea5760cc9278f"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "2ef18a3cc45056e40e9831ea982aebbc"
server_key_iv: "730525899487d0917fc20141786e08d4"
server_key_version: "encryptionkey@0"
I20260928 22:54:11.614269 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "36c03d23300f439da64ea5760cc9278f"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "2ef18a3cc45056e40e9831ea982aebbc"
server_key_iv: "730525899487d0917fc20141786e08d4"
server_key_version: "encryptionkey@0"
I20260928 22:54:11.615545 22757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:54:11.616228 22772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:11.616420 22757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:11.616520 22757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "36c03d23300f439da64ea5760cc9278f"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "2ef18a3cc45056e40e9831ea982aebbc"
server_key_iv: "730525899487d0917fc20141786e08d4"
server_key_version: "encryptionkey@0"
I20260928 22:54:11.616576 22757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:11.625267 22757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:11.627182 22757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:11.627298 22757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:11.627475 22757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:11.627710 22757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:11.627740 22757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:11.627761 22757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:11.627774 22757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:11.633354 22757 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:36055
I20260928 22:54:11.633426 22885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:36055 every 8 connection(s)
I20260928 22:54:11.633802 22757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:54:11 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636051, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:54:11.642602 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:11.634530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:34645 (local address 127.14.14.190:35131)
0928 22:54:11.634610 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:11.634612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:11.634771 (+ 159us) server_negotiation.cc:407] Connection header received
0928 22:54:11.635267 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:11.635268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:11.635278 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:11.635337 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:11.636270 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:11.636780 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:11.637826 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:11.638027 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:11.638088 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:11.640195 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:11.640197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:11.640197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:11.640203 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:11.641433 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:11.641847 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:11.641847 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:11.641848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:11.641898 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:11.642109 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:11.642109 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:11.642109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:11.642200 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:11.642244 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:11.642481 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 22:54:11.642538 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 22:54:11.642791 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:11.634482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:34645)
0928 22:54:11.634647 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:11.634681 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:11.635154 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:11.635408 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:11.635411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:11.635955 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:11.636160 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:11.636165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:11.636862 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:11.636863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:11.637727 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:11.637730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:11.638118 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:11.638119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:11.638214 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:11.638744 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:11.638752 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:11.640104 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:11.641513 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:11.641519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:11.641520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:11.641781 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:11.641958 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:11.641959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:11.641959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:11.642055 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:11.642250 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:11.642254 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:11.642361 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 22:54:11.642442 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 22:54:11.642554 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260928 22:54:11.643093 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 22757
I20260928 22:54:11.643225 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:54:11.643421 14394 external_mini_cluster.cc:1428] Setting key 04dba016ee7a7cce24b21bc0b200c196
I20260928 22:54:11.643435 22886 heartbeater.cc:347] Connected to a master server at 127.14.14.190:35131
I20260928 22:54:11.643540 22886 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:11.643772 22886 heartbeater.cc:511] Master 127.14.14.190:35131 requested a full tablet report, sending...
I20260928 22:54:11.644138 22561 ts_manager.cc:194] Registered new tserver with Master: 36c03d23300f439da64ea5760cc9278f (127.14.14.130:36055)
I20260928 22:54:11.644534 22561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:34645
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:54:11.669353 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:35131
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:11.750262 22891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:11.750510 22891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:11.750551 22891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:11.751710 22891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:11.751781 22891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:11.751828 22891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:54:11.753336 22891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:35131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:11.753616 22891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:11.753857 22891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:11.758764 22899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:11.758795 22897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:11.758813 22896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:11.758991 22891 server_base.cc:1034] running on GCE node
I20260928 22:54:11.759200 22891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:11.759449 22891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:11.760633 22891 hybrid_clock.cc:648] HybridClock initialized: now 1790636051760616 us; error 28 us; skew 500 ppm
Sep 28 22:54:11 dist-test-slave-kzfr krb5kdc[22516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636051, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:11.763121 22891 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:54:11.763629 22891 webserver.cc:460] Webserver started at http://127.14.14.131:34237/ using document root <none> and password file <none>
I20260928 22:54:11.763861 22891 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:11.763931 22891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:11.764086 22891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:11.765120 22891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "2b6bb05bd1564afeb79007954f84d5a1"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "a2f845117f425dffc3c011724b33febc"
server_key_iv: "a82b98c325ddf243c93823a3eab2941e"
server_key_version: "encryptionkey@0"
I20260928 22:54:11.765491 22891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "2b6bb05bd1564afeb79007954f84d5a1"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "a2f845117f425dffc3c011724b33febc"
server_key_iv: "a82b98c325ddf243c93823a3eab2941e"
server_key_version: "encryptionkey@0"
I20260928 22:54:11.766835 22891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:54:11.767546 22906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:11.767725 22891 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:54:11.767824 22891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "2b6bb05bd1564afeb79007954f84d5a1"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "a2f845117f425dffc3c011724b33febc"
server_key_iv: "a82b98c325ddf243c93823a3eab2941e"
server_key_version: "encryptionkey@0"
I20260928 22:54:11.767903 22891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:11.778183 22891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:11.780066 22891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:11.780226 22891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:11.780462 22891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:11.780777 22891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:11.780838 22891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:11.780897 22891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:11.780926 22891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:11.787303 22891 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:38241
I20260928 22:54:11.787348 23019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:38241 every 8 connection(s)
I20260928 22:54:11.787676 22891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:54:11 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636051, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:54:11.793793 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 22891
I20260928 22:54:11.793912 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:54:11.794142 14394 external_mini_cluster.cc:1428] Setting key 88d26f3b556877d5e9ea3b586119d496
I20260928 22:54:11.797206 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:11.788396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:36625 (local address 127.14.14.190:35131)
0928 22:54:11.788485 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:11.788487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:11.788688 (+ 201us) server_negotiation.cc:407] Connection header received
0928 22:54:11.789189 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:11.789190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:11.789201 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:11.789265 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:11.790315 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:11.790809 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:11.791767 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:11.791946 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:11.792010 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:11.794474 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:11.794476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:11.794476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:11.794484 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:11.795958 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:11.796353 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:11.796354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:11.796354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:11.796414 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:11.796638 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:11.796638 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:11.796638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:11.796741 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:11.796791 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:11.797065 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 22:54:11.797142 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:54:11.797334 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:11.788374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:36625)
0928 22:54:11.788539 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:11.788595 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:11.789084 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:11.789321 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:11.789325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:11.790021 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:11.790219 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:11.790225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:11.790892 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:11.790892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:11.791687 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:11.791689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:11.792027 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:11.792028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:11.792129 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:11.792650 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:11.792658 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:11.794328 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:11.796046 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:11.796052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:11.796054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:11.796274 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:11.796481 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:11.796481 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:11.796482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:11.796581 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:11.796795 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:11.796798 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:11.796912 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 22:54:11.797003 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 22:54:11.797122 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260928 22:54:11.797785 23020 heartbeater.cc:347] Connected to a master server at 127.14.14.190:35131
I20260928 22:54:11.797883 23020 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:11.798030 23020 heartbeater.cc:511] Master 127.14.14.190:35131 requested a full tablet report, sending...
I20260928 22:54:11.798412 22561 ts_manager.cc:194] Registered new tserver with Master: 2b6bb05bd1564afeb79007954f84d5a1 (127.14.14.131:38241)
I20260928 22:54:11.798767 22561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:36625
I20260928 22:54:11.806360 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.129.keytab.
Entry for principal oryx/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.129.keytab.
WARNING: no policy specified for oryx/127.14.14.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.130.keytab.
Entry for principal oryx/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.130.keytab.
WARNING: no policy specified for oryx/127.14.14.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.131.keytab.
Entry for principal oryx/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.131.keytab.
I20260928 22:54:11.880326 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 22623
I20260928 22:54:11.884812 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 22757
I20260928 22:54:11.889024 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 22891
I20260928 22:54:11.893411 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 22529
I20260928 22:54:11.898350 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:35131
--webserver_interface=127.14.14.190
--webserver_port=42457
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:35131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:11.979293 23029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:11.979544 23029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:11.979583 23029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:11.980707 23029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:54:11.980777 23029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:11.980804 23029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:11.980834 23029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:54:11.980856 23029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:54:11.982388 23029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:35131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:35131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=42457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:11.982661 23029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:11.982926 23029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:11.987728 23037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:11.987728 23035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:11.987736 23034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:11.987934 23029 server_base.cc:1034] running on GCE node
I20260928 22:54:11.988242 23029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:11.988487 23029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:11.989651 23029 hybrid_clock.cc:648] HybridClock initialized: now 1790636051989638 us; error 28 us; skew 500 ppm
Sep 28 22:54:11 dist-test-slave-kzfr krb5kdc[22516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636051, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:11.992318 23029 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:54:11.992843 23029 webserver.cc:460] Webserver started at http://127.14.14.190:42457/ using document root <none> and password file <none>
I20260928 22:54:11.993096 23029 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:11.993155 23029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:11.994554 23029 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:11.995208 23044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:11.995404 23029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:11.995483 23029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "67555a2cc798471eb687be38a1f4f206"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "df9492a170c834500e2694ae56b88112"
server_key_iv: "f8adada12bb1003a3c0b53ed3fd4f1c6"
server_key_version: "encryptionkey@0"
I20260928 22:54:11.995798 23029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:12.023465 23029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:12.025918 23029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:12.026034 23029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:12.029815 23029 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:35131
I20260928 22:54:12.029896 23096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:35131 every 8 connection(s)
I20260928 22:54:12.030273 23029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:54:12.031746 23097 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:54:12.032403 23097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206: Bootstrap starting.
I20260928 22:54:12.032826 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 23029
I20260928 22:54:12.033172 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:43771
--local_ip_for_outbound_sockets=127.14.14.129
--tserver_master_addrs=127.14.14.190:35131
--webserver_port=38353
--webserver_interface=127.14.14.129
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.14.129
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:54:12.041042 23097 log.cc:824] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:12.042913 23097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:54:12.043222 23097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206: Bootstrap complete.
I20260928 22:54:12.045447 23097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67555a2cc798471eb687be38a1f4f206" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35131 } }
I20260928 22:54:12.045766 23097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67555a2cc798471eb687be38a1f4f206, State: Initialized, Role: FOLLOWER
I20260928 22:54:12.045982 23097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67555a2cc798471eb687be38a1f4f206" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35131 } }
I20260928 22:54:12.046082 23097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:54:12.046136 23097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:54:12.046185 23097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:54:12.047216 23097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67555a2cc798471eb687be38a1f4f206" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35131 } }
I20260928 22:54:12.047343 23097 leader_election.cc:302] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67555a2cc798471eb687be38a1f4f206; no voters:
I20260928 22:54:12.047573 23097 leader_election.cc:288] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 22:54:12.047616 23101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:54:12.047739 23101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [term 2 LEADER]: Becoming Leader. State: Replica: 67555a2cc798471eb687be38a1f4f206, State: Running, Role: LEADER
I20260928 22:54:12.047842 23097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:54:12.047848 23101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67555a2cc798471eb687be38a1f4f206" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35131 } }
I20260928 22:54:12.048136 23101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67555a2cc798471eb687be38a1f4f206. Latest consensus state: current_term: 2 leader_uuid: "67555a2cc798471eb687be38a1f4f206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67555a2cc798471eb687be38a1f4f206" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35131 } } }
I20260928 22:54:12.048204 23101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:12.048343 23103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "67555a2cc798471eb687be38a1f4f206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67555a2cc798471eb687be38a1f4f206" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 35131 } } }
I20260928 22:54:12.048417 23103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206 [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:12.048699 23106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:54:12.049235 23106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:54:12.049389 23106 catalog_manager.cc:1259] Loaded cluster ID: dc2bd9b135dd4f01a0a3e3b002209b5a
I20260928 22:54:12.049677 23106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:54:12.050539 23106 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:54:12.050738 23106 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 67555a2cc798471eb687be38a1f4f206: Loaded TSK: 0
I20260928 22:54:12.050966 23106 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:54:12.133330 23099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:12.133566 23099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:12.133626 23099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:12.134797 23099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:12.134867 23099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:12.134913 23099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:54:12.136381 23099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.129.keytab
--principal=oryx/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:43771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=38353
--tserver_master_addrs=127.14.14.190:35131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:12.136665 23099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:12.136893 23099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:12.141974 23126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:12.142102 23124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:12.142006 23123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:12.142313 23099 server_base.cc:1034] running on GCE node
I20260928 22:54:12.142539 23099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:12.142827 23099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:12.144016 23099 hybrid_clock.cc:648] HybridClock initialized: now 1790636052143997 us; error 32 us; skew 500 ppm
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636052, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:12.147001 23099 init.cc:377] Logged in from keytab as oryx/127.14.14.129@KRBTEST.COM (short username oryx)
I20260928 22:54:12.147578 23099 webserver.cc:460] Webserver started at http://127.14.14.129:38353/ using document root <none> and password file <none>
I20260928 22:54:12.147814 23099 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:12.147898 23099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:12.149441 23099 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:12.150117 23133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:12.150334 23099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:54:12.150431 23099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "79fc4b0ee59642f8b765ed4afcee4ecd"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "4e2f5a8433a1809f0ea628b62eb003e1"
server_key_iv: "5b7f062bfa30fea293416bd467f77905"
server_key_version: "encryptionkey@0"
I20260928 22:54:12.150795 23099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:12.170800 23099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:12.172946 23099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:12.173110 23099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:12.173332 23099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:12.173740 23099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:12.173796 23099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:12.173840 23099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:12.173867 23099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:12.180302 23099 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:43771
I20260928 22:54:12.180351 23246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:43771 every 8 connection(s)
I20260928 22:54:12.180678 23099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:54:12.188894 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 23099
I20260928 22:54:12.189229 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:36055
--local_ip_for_outbound_sockets=127.14.14.130
--tserver_master_addrs=127.14.14.190:35131
--webserver_port=32905
--webserver_interface=127.14.14.130
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.14.130
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:12.191111 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:12.181412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:49187)
0928 22:54:12.181633 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:12.181682 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:12.182116 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:12.183680 (+ 1564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:12.183684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:12.184350 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:12.184557 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.184561 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.186185 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.186185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.186945 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.186947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.187760 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.187760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.187864 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.188374 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:12.188382 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:12.190977 (+ 2595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
W20260928 22:54:12.191270 23250 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:12.181464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:49187 (local address 127.14.14.190:35131)
0928 22:54:12.183268 (+ 1804us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:12.183277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:12.183313 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:54:12.183436 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:12.183441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:12.183494 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:12.183633 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:12.184673 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.186095 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.187040 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.187665 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.187769 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.191106 (+ 3337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:49187: BlockingRecv error: recv got EOF from 127.14.14.129:49187 (error 108)
Metrics: {"server-negotiator.queue_time_us":1726,"thread_start_us":96,"threads_started":1}
W20260928 22:54:12.191630 23247 heartbeater.cc:644] Failed to heartbeat to 127.14.14.190:35131 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.14.190:35131: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:12.196139 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:12.191846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:50147)
0928 22:54:12.191954 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:12.191972 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:12.192024 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:12.192156 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:12.192157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:12.192282 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:12.192323 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.192324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.192972 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.192973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.193525 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.193526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.193932 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.193933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.194008 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.194457 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:12.194462 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:12.196056 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260928 22:54:12.196192 23250 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:12.191909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:50147 (local address 127.14.14.190:35131)
0928 22:54:12.191944 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:12.191947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:12.192009 (+ 62us) server_negotiation.cc:407] Connection header received
0928 22:54:12.192081 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:12.192082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:12.192093 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:12.192144 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:12.192406 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.192873 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.193657 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.193848 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.193932 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.196128 (+ 2196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:50147: BlockingRecv error: recv got EOF from 127.14.14.129:50147 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:12.200927 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:12.196645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:39671)
0928 22:54:12.196720 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:12.196731 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:12.196776 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:12.196888 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:12.196889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:12.197010 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:12.197033 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.197034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.197675 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.197676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.198236 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.198238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.198693 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.198694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.198749 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.199181 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:12.199186 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:12.200837 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 22:54:12.200984 23250 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:12.196687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:39671 (local address 127.14.14.190:35131)
0928 22:54:12.196723 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:12.196725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:12.196762 (+ 37us) server_negotiation.cc:407] Connection header received
0928 22:54:12.196831 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:12.196832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:12.196841 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:12.196888 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:12.197114 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.197556 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.198318 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.198611 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.198690 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.200913 (+ 2223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:39671: BlockingRecv error: recv got EOF from 127.14.14.129:39671 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 22:54:12.201272 23247 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:54:12.275525 23251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:12.275725 23251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:12.275750 23251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:12.276853 23251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:12.276901 23251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:12.276926 23251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:54:12.278339 23251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.130.keytab
--principal=oryx/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:36055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=32905
--tserver_master_addrs=127.14.14.190:35131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:12.278527 23251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:12.278702 23251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:12.283819 23257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:12.283871 23251 server_base.cc:1034] running on GCE node
W20260928 22:54:12.283998 23256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:12.283996 23259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:12.284348 23251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:12.284592 23251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:12.285754 23251 hybrid_clock.cc:648] HybridClock initialized: now 1790636052285744 us; error 31 us; skew 500 ppm
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636052, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:12.288267 23251 init.cc:377] Logged in from keytab as oryx/127.14.14.130@KRBTEST.COM (short username oryx)
I20260928 22:54:12.288663 23251 webserver.cc:460] Webserver started at http://127.14.14.130:32905/ using document root <none> and password file <none>
I20260928 22:54:12.288841 23251 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:12.288894 23251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:12.290273 23251 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:12.290849 23266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:12.291077 23251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:12.291186 23251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "36c03d23300f439da64ea5760cc9278f"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "2ef18a3cc45056e40e9831ea982aebbc"
server_key_iv: "730525899487d0917fc20141786e08d4"
server_key_version: "encryptionkey@0"
I20260928 22:54:12.291532 23251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:12.301525 23251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:12.303840 23251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:12.304004 23251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:12.304250 23251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:12.304545 23251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:12.304606 23251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:12.304646 23251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:12.304670 23251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:12.310606 23251 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:36055
I20260928 22:54:12.310650 23379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:36055 every 8 connection(s)
I20260928 22:54:12.311008 23251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:54:12.316121 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 23251
I20260928 22:54:12.316431 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:38241
--local_ip_for_outbound_sockets=127.14.14.131
--tserver_master_addrs=127.14.14.190:35131
--webserver_port=34237
--webserver_interface=127.14.14.131
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.14.131
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:12.318619 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:12.311782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:55711)
0928 22:54:12.311943 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:12.311977 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:12.312481 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:12.312745 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:12.312748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:12.313311 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:12.313500 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.313504 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.314268 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.314269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.315058 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.315060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.315430 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.315432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.315553 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.316045 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:12.316055 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:12.318478 (+ 2423us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
W20260928 22:54:12.318675 23250 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:12.311778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:55711 (local address 127.14.14.190:35131)
0928 22:54:12.311853 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:12.311855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:12.312069 (+ 214us) server_negotiation.cc:407] Connection header received
0928 22:54:12.312594 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:12.312596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:12.312611 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:12.312669 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:12.313641 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.314166 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.315155 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.315344 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.315424 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.318604 (+ 3180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:55711: BlockingRecv error: recv got EOF from 127.14.14.130:55711 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 22:54:12.319203 23380 heartbeater.cc:644] Failed to heartbeat to 127.14.14.190:35131 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.14.190:35131: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:12.323837 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:12.319449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:49333)
0928 22:54:12.319498 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:12.319507 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:12.319569 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:12.319716 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:12.319717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:12.319871 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:12.319928 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.319930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.320564 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.320565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.321070 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.321071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.321550 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.321550 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.321680 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.322106 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:12.322111 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:12.323747 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 22:54:12.323889 23250 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:12.319496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:49333 (local address 127.14.14.190:35131)
0928 22:54:12.319562 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:12.319564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:12.319573 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:54:12.319633 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:12.319634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:12.319643 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:12.319714 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:12.319992 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.320465 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.321186 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.321470 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.321538 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.323823 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:49333: BlockingRecv error: recv got EOF from 127.14.14.130:49333 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:12.328250 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:12.324236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:60377)
0928 22:54:12.324284 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:12.324294 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:12.324356 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:12.324461 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:12.324461 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:12.324592 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:12.324614 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.324615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.325228 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.325229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.325800 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.325802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.326156 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.326157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.326208 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.326624 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:12.326628 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:12.328176 (+ 1548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:54:12.328303 23250 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:12.324306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:60377 (local address 127.14.14.190:35131)
0928 22:54:12.324363 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:12.324365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:12.324375 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:54:12.324407 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:12.324407 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:12.324416 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:12.324460 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:12.324691 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.325147 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.325892 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.326086 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.326159 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.328242 (+ 2083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:60377: BlockingRecv error: recv got EOF from 127.14.14.130:60377 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 22:54:12.328514 23380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:54:12.400079 23383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:12.400285 23383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:12.400308 23383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:12.401562 23383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:12.401660 23383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:12.401713 23383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:54:12.403137 23383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/krb5kdc/oryx_127.14.14.131.keytab
--principal=oryx/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:38241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=34237
--tserver_master_addrs=127.14.14.190:35131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:12.403376 23383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:12.403613 23383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:12.408812 23388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:12.408846 23391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:12.408812 23389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:12.409080 23383 server_base.cc:1034] running on GCE node
I20260928 22:54:12.409324 23383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:12.409642 23383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:12.410810 23383 hybrid_clock.cc:648] HybridClock initialized: now 1790636052410794 us; error 33 us; skew 500 ppm
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636052, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:12.413338 23383 init.cc:377] Logged in from keytab as oryx/127.14.14.131@KRBTEST.COM (short username oryx)
I20260928 22:54:12.413818 23383 webserver.cc:460] Webserver started at http://127.14.14.131:34237/ using document root <none> and password file <none>
I20260928 22:54:12.414005 23383 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:12.414058 23383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:12.415320 23383 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:12.415870 23398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:12.416035 23383 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:12.416132 23383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "2b6bb05bd1564afeb79007954f84d5a1"
format_stamp: "Formatted at 2026-09-28 22:54:11 on dist-test-slave-kzfr"
server_key: "a2f845117f425dffc3c011724b33febc"
server_key_iv: "a82b98c325ddf243c93823a3eab2941e"
server_key_version: "encryptionkey@0"
I20260928 22:54:12.416452 23383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:12.431000 23383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:12.433045 23383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:12.433177 23383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:12.433357 23383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:12.433766 23383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:12.433804 23383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:12.433828 23383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:12.433885 23383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:12.439564 23383 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:38241
I20260928 22:54:12.439647 23511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:38241 every 8 connection(s)
I20260928 22:54:12.440007 23383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:54:12.441823 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 23383
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 22:54:12.448998 23250 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:12.440792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:41843 (local address 127.14.14.190:35131)
0928 22:54:12.440866 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:12.440880 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:12.441084 (+ 204us) server_negotiation.cc:407] Connection header received
0928 22:54:12.441685 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:12.441687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:12.441700 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:12.441761 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:12.443008 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.443615 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.445003 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.445327 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.445409 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.448911 (+ 3502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:41843: BlockingRecv error: recv got EOF from 127.14.14.131:41843 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":44}
I20260928 22:54:12.449033 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:12.440792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:41843)
0928 22:54:12.440952 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:12.440994 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:12.441546 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:12.441837 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:12.441840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:12.442615 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:12.442913 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.442920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.443706 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.443707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.444896 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.444899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.445427 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.445428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.445610 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.446054 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:12.446065 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:12.448813 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
W20260928 22:54:12.449260 23512 heartbeater.cc:644] Failed to heartbeat to 127.14.14.190:35131 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.14.190:35131: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
I20260928 22:54:12.450603 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:12.442346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.14.14.190:35131)
0928 22:54:12.442433 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:12.442442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:12.442483 (+ 41us) server_negotiation.cc:407] Connection header received
0928 22:54:12.442579 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:12.442580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:12.442589 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:12.442648 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:12.443389 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.443956 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.445003 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.445327 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.445399 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.446851 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:12.446855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:12.446856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:12.446868 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:12.448993 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:12.449413 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:12.449421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:12.449422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:12.449552 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:12.449830 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:12.449831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:12.449832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:12.450149 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:12.450252 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:12.450394 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 22:54:12.450486 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":68,"thread_start_us":43,"threads_started":1}
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:12.454015 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:12.449465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:43253)
0928 22:54:12.449527 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:12.449539 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:12.449577 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:12.449747 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:12.449748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:12.449901 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:12.449949 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.449951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.450811 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.450812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.451458 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.451461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.451816 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.451817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.451882 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.452236 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:12.452240 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:12.453938 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 22:54:12.454062 23250 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:12.449557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:43253 (local address 127.14.14.190:35131)
0928 22:54:12.449641 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:12.449643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:12.449650 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:54:12.449669 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:12.449669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:12.449680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:12.449735 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:12.450064 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.450743 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.451532 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.451733 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.451817 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.454003 (+ 2186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:43253: BlockingRecv error: recv got EOF from 127.14.14.131:43253 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:12 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:12.458098 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:12.454296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:42617)
0928 22:54:12.454326 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:12.454336 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:12.454370 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:12.454494 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:12.454495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:12.454623 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:12.454646 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.454647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.455252 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.455253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.455758 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:12.455759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.456145 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.456146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:12.456198 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.456507 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:12.456510 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:12.458058 (+ 1548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 22:54:12.458204 23250 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:12.454361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:42617 (local address 127.14.14.190:35131)
0928 22:54:12.454400 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:12.454402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:12.454410 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:54:12.454429 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:12.454430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:12.454437 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:12.454492 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:12.454730 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.455172 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.455860 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:12.456055 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:12.456144 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:12.458140 (+ 1996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:42617: BlockingRecv error: recv got EOF from 127.14.14.131:42617 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 22:54:12.458261 23512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 22:54:13 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:13 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:13.207310 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:13.201780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:39347)
0928 22:54:13.202005 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:13.202016 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:13.202074 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:13.202262 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:13.202262 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:13.202455 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:13.202504 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:13.202506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:13.203261 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:13.203262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:13.204117 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:13.204120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:13.204477 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:13.204497 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:13.204547 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:13.205064 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:13.205069 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:13.207244 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
W20260928 22:54:13.207410 23518 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:13.201868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:39347 (local address 127.14.14.190:35131)
0928 22:54:13.202117 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:13.202120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:13.202129 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:54:13.202155 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:13.202156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:13.202167 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:13.202243 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:13.202596 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:13.203152 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:13.204189 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:13.204388 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:13.204439 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:13.207337 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:39347: BlockingRecv error: recv got EOF from 127.14.14.129:39347 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":181,"threads_started":1}
Sep 28 22:54:13 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:13 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:13.334837 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:13.328969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:57767)
0928 22:54:13.329114 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:13.329128 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:13.329196 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:13.329380 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:13.329381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:13.329650 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:13.329702 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:13.329705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:13.330523 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:13.330524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:13.331164 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:13.331166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:13.331611 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:13.331612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:13.331681 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:13.332217 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:13.332221 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:13.334714 (+ 2493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
W20260928 22:54:13.335094 23518 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:13.329099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:57767 (local address 127.14.14.190:35131)
0928 22:54:13.329176 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:13.329179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:13.329190 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:54:13.329268 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:13.329269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:13.329283 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:13.329356 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:13.329791 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:13.330420 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:13.331279 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:13.331494 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:13.331590 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:13.334831 (+ 3241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:57767: BlockingRecv error: recv got EOF from 127.14.14.130:57767 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 22:54:13 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:13 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:13.463512 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:13.458727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:53531)
0928 22:54:13.458897 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:13.458908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:13.459015 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:13.459145 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:13.459146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:13.459306 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:13.459354 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:13.459355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:13.460040 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:13.460041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:13.460649 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:13.460651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:13.461020 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:13.461021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:13.461065 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:13.461522 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:13.461528 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:13.463427 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
W20260928 22:54:13.463616 23518 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:13.458792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:53531 (local address 127.14.14.190:35131)
0928 22:54:13.458934 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:13.458936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:13.459005 (+ 69us) server_negotiation.cc:407] Connection header received
0928 22:54:13.459073 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:13.459073 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:13.459085 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:13.459128 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:13.459450 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:13.459937 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:13.460731 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:13.460919 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:13.460968 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:13.463531 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:53531: BlockingRecv error: recv got EOF from 127.14.14.131:53531 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 28 22:54:14 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:14 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:14.219074 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:14.207791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:36925)
0928 22:54:14.207919 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:14.207935 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:14.207983 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:14.208182 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:14.208182 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:14.208334 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:14.208383 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:14.208384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:14.209043 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:14.209044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:14.209563 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:14.209564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:14.209962 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:14.209963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:14.210007 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:14.216883 (+ 6876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:14.216888 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:14.219013 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
W20260928 22:54:14.219311 23522 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:14.207927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:36925 (local address 127.14.14.190:35131)
0928 22:54:14.208065 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:14.208067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:14.208075 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:54:14.208096 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:14.208097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:14.208106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:14.208168 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:14.208449 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:14.208969 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:14.209697 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:14.209895 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:14.209951 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:14.219231 (+ 9280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:36925: BlockingRecv error: recv got EOF from 127.14.14.129:36925 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
Sep 28 22:54:14 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:14 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:14.340543 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:14.335471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:41917)
0928 22:54:14.335723 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:14.335737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:14.335838 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:14.336193 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:14.336194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:14.336399 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:14.336458 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:14.336460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:14.337144 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:14.337145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:14.337713 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:14.337715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:14.338105 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:14.338107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:14.338165 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:14.338668 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:14.338673 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:14.340444 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":107,"threads_started":1}
W20260928 22:54:14.340590 23522 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:14.335592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:41917 (local address 127.14.14.190:35131)
0928 22:54:14.335708 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:14.335711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:14.335822 (+ 111us) server_negotiation.cc:407] Connection header received
0928 22:54:14.336095 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:14.336096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:14.336112 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:14.336182 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:14.336546 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:14.337053 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:14.337807 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:14.338013 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:14.338104 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:14.340525 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:41917: BlockingRecv error: recv got EOF from 127.14.14.130:41917 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 22:54:14 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:14 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:14.469167 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:14.464174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:55633)
0928 22:54:14.464390 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:14.464402 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:14.464471 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:14.464640 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:14.464641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:14.464805 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:14.464857 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:14.464858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:14.465848 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:14.465849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:14.466439 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:14.466442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:14.466811 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:14.466811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:14.466850 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:14.467320 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:14.467324 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:14.469095 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":109,"threads_started":1}
W20260928 22:54:14.469272 23522 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:14.464366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:55633 (local address 127.14.14.190:35131)
0928 22:54:14.464443 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:14.464445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:14.464458 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:54:14.464543 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:14.464544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:14.464557 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:14.464612 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:14.464930 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:14.465761 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:14.466540 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:14.466740 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:14.466798 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:14.469200 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:55633: BlockingRecv error: recv got EOF from 127.14.14.131:55633 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 22:54:15 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:15 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:15.224527 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:15.219648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:54525)
0928 22:54:15.219935 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:15.219949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:15.220022 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:15.220177 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:15.220178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:15.220342 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:15.220394 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:15.220396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:15.221083 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:15.221084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:15.221617 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:15.221618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:15.222008 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:15.222009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:15.222065 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:15.222546 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:15.222550 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:15.224433 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":95,"threads_started":1}
W20260928 22:54:15.224597 23526 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:15.219742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:54525 (local address 127.14.14.190:35131)
0928 22:54:15.219911 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:15.219914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:15.220045 (+ 131us) server_negotiation.cc:407] Connection header received
0928 22:54:15.220078 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:15.220079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:15.220091 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:15.220158 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:15.220463 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:15.221013 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:15.221717 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:15.221920 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:15.222007 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:15.224525 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:54525: BlockingRecv error: recv got EOF from 127.14.14.129:54525 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
Sep 28 22:54:15 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:15 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:15.346460 23527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:15.341240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:52465)
0928 22:54:15.341402 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:15.341413 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:15.341474 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:15.341697 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:15.341698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:15.341854 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:15.341903 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:15.341905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:15.342735 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:15.342736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:15.343241 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:15.343242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:15.343653 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:15.343654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:15.343705 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:15.344207 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:15.344213 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:15.346398 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
W20260928 22:54:15.346652 23526 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:15.341399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:52465 (local address 127.14.14.190:35131)
0928 22:54:15.341487 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:15.341489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:15.341530 (+ 41us) server_negotiation.cc:407] Connection header received
0928 22:54:15.341559 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:15.341560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:15.341571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:15.341672 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:15.342079 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:15.342657 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:15.343327 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:15.343551 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:15.343616 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:15.346583 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:52465: BlockingRecv error: recv got EOF from 127.14.14.130:52465 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 22:54:15 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:15 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:15.474627 23528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:15.469749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:33443)
0928 22:54:15.469967 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:15.469979 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:15.470048 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:15.470210 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:15.470211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:15.470392 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:15.470440 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:15.470442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:15.471122 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:15.471123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:15.471673 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:15.471674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:15.472049 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:15.472050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:15.472106 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:15.472626 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:15.472631 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:15.474528 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":135,"threads_started":1}
W20260928 22:54:15.474670 23526 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:15.469876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:33443 (local address 127.14.14.190:35131)
0928 22:54:15.469950 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:15.469952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:15.470046 (+ 94us) server_negotiation.cc:407] Connection header received
0928 22:54:15.470116 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:15.470117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:15.470131 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:15.470186 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:15.470524 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:15.471035 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:15.471765 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:15.471973 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:15.472052 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:15.474608 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:33443: BlockingRecv error: recv got EOF from 127.14.14.131:33443 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 22:54:16 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:16 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:16.230422 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:16.225117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:43969)
0928 22:54:16.225280 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:16.225292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:16.225347 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:16.225510 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:16.225511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:16.225707 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:16.225756 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:16.225758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:16.226498 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:16.226499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:16.227132 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:16.227135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:16.227698 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:16.227699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:16.227743 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:16.228258 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:16.228263 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:16.230309 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
W20260928 22:54:16.230509 23530 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:16.225210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:43969 (local address 127.14.14.190:35131)
0928 22:54:16.225359 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:16.225361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:16.225387 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:54:16.225411 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:16.225411 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:16.225421 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:16.225474 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:16.225856 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:16.226422 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:16.227218 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:16.227617 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:16.227690 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:16.230400 (+ 2710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:43969: BlockingRecv error: recv got EOF from 127.14.14.129:43969 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
Sep 28 22:54:16 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:16 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:16.352289 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:16.347057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:33367)
0928 22:54:16.347271 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:16.347303 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:16.347394 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:16.347617 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:16.347618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:16.347865 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:16.347922 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:16.347924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:16.348665 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:16.348666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:16.349405 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:16.349408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:16.349855 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:16.349856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:16.349913 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:16.350381 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:16.350386 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:16.352230 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":66,"threads_started":1}
W20260928 22:54:16.352447 23530 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:16.347139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:33367 (local address 127.14.14.190:35131)
0928 22:54:16.347195 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:16.347197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:16.347388 (+ 191us) server_negotiation.cc:407] Connection header received
0928 22:54:16.347488 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:16.347489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:16.347505 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:16.347588 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:16.348013 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:16.348561 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:16.349488 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:16.349739 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:16.349798 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:16.352359 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:33367: BlockingRecv error: recv got EOF from 127.14.14.130:33367 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 22:54:16 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:16 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:16.480727 23532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:16.475234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:57373)
0928 22:54:16.475476 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:16.475498 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:16.475578 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:16.475759 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:16.475761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:16.475959 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:16.476016 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:16.476018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:16.476923 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:16.476925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:16.477552 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:16.477554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:16.478029 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:16.478030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:16.478095 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:16.478657 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:16.478663 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:16.480620 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":112,"threads_started":1}
W20260928 22:54:16.480790 23530 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:16.475363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:57373 (local address 127.14.14.190:35131)
0928 22:54:16.475456 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:16.475459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:16.475573 (+ 114us) server_negotiation.cc:407] Connection header received
0928 22:54:16.475651 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:16.475652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:16.475667 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:16.475751 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:16.476127 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:16.476832 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:16.477704 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:16.477922 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:16.478028 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:16.480720 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:57373: BlockingRecv error: recv got EOF from 127.14.14.131:57373 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 22:54:17 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:17 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:17.235605 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:17.230955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:52487)
0928 22:54:17.231086 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:17.231097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:17.231176 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:17.231348 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:17.231348 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:17.231518 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:17.231566 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:17.231567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:17.232205 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:17.232205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:17.232735 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:17.232737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:17.233099 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:17.233100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:17.233140 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:17.233724 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:17.233730 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:17.235547 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
W20260928 22:54:17.235782 23534 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:17.231012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:52487 (local address 127.14.14.190:35131)
0928 22:54:17.231143 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:17.231145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:17.231155 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:54:17.231254 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:17.231254 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:17.231267 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:17.231343 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:17.231632 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:17.232133 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:17.232821 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:17.233025 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:17.233093 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:17.235694 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:52487: BlockingRecv error: recv got EOF from 127.14.14.129:52487 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
Sep 28 22:54:17 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:17 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:17.357535 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:17.352793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:43773)
0928 22:54:17.352952 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:17.352964 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:17.353070 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:17.353258 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:17.353259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:17.353443 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:17.353494 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:17.353496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:17.354219 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:17.354220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:17.354758 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:17.354760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:17.355133 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:17.355134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:17.355195 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:17.355680 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:17.355684 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:17.357437 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
W20260928 22:54:17.357607 23534 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:17.352953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:43773 (local address 127.14.14.190:35131)
0928 22:54:17.353033 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:17.353036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:17.353044 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:54:17.353153 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:17.353153 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:17.353166 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:17.353243 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:17.353615 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:17.354114 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:17.354852 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:17.355046 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:17.355133 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:17.357519 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:43773: BlockingRecv error: recv got EOF from 127.14.14.130:43773 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 22:54:17 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:17 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:17.486343 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:17.481384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:38663)
0928 22:54:17.481540 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:17.481552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:17.481639 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:17.481782 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:17.481783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:17.481948 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:17.481999 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:17.482000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:17.482638 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:17.482639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:17.483277 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:17.483280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:17.483788 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:17.483789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:17.483856 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:17.484379 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:17.484384 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:17.486243 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
W20260928 22:54:17.486394 23534 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:17.481442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:38663 (local address 127.14.14.190:35131)
0928 22:54:17.481514 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:17.481516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:17.481641 (+ 125us) server_negotiation.cc:407] Connection header received
0928 22:54:17.481703 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:17.481703 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:17.481718 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:17.481761 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:17.482063 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:17.482563 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:17.483393 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:17.483705 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:17.483774 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:17.486328 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:38663: BlockingRecv error: recv got EOF from 127.14.14.131:38663 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 22:54:18 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:18 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:18.240808 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:18.236121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:56609)
0928 22:54:18.236330 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:18.236358 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:18.236422 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:18.236573 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:18.236574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:18.236733 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:18.236782 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:18.236784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:18.237480 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:18.237481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:18.238035 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:18.238036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:18.238371 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:18.238372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:18.238411 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:18.238935 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:18.238941 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:18.240748 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":108,"threads_started":1}
W20260928 22:54:18.241016 23538 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:18.236193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:56609 (local address 127.14.14.190:35131)
0928 22:54:18.236331 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:18.236333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:18.236412 (+ 79us) server_negotiation.cc:407] Connection header received
0928 22:54:18.236491 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:18.236492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:18.236504 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:18.236554 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:18.236876 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:18.237410 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:18.238113 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:18.238304 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:18.238358 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:18.240909 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:56609: BlockingRecv error: recv got EOF from 127.14.14.129:56609 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
Sep 28 22:54:18 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:18 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:18.362779 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:18.358109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:45803)
0928 22:54:18.358254 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:18.358265 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:18.358314 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:18.358446 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:18.358447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:18.358605 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:18.358656 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:18.358657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:18.359421 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:18.359422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:18.359933 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:18.359934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:18.360300 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:18.360301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:18.360339 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:18.360860 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:18.360865 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:18.362722 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
W20260928 22:54:18.362979 23538 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:18.358234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:45803 (local address 127.14.14.190:35131)
0928 22:54:18.358301 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:18.358303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:18.358313 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:54:18.358360 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:18.358361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:18.358372 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:18.358423 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:18.358778 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:18.359345 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:18.360016 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:18.360229 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:18.360289 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:18.362885 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:45803: BlockingRecv error: recv got EOF from 127.14.14.130:45803 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 22:54:18 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:18 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:18.492406 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:18.487039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:43361)
0928 22:54:18.487261 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:18.487274 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:18.487518 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:18.487750 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:18.487751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:18.488003 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:18.488061 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:18.488063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:18.488814 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:18.488816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:18.489367 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:18.489369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:18.489855 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:18.489856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:18.489916 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:18.490405 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:18.490409 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:18.492332 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":71,"threads_started":1}
W20260928 22:54:18.492554 23538 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:18.487297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:43361 (local address 127.14.14.190:35131)
0928 22:54:18.487412 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:18.487414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:18.487487 (+ 73us) server_negotiation.cc:407] Connection header received
0928 22:54:18.487648 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:18.487649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:18.487668 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:18.487741 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:18.488213 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:18.488716 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:18.489457 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:18.489758 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:18.489851 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:18.492459 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:43361: BlockingRecv error: recv got EOF from 127.14.14.131:43361 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 28 22:54:19 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:19 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:19.251462 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:19.241445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:43987)
0928 22:54:19.241635 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:19.241646 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:19.241700 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:19.241861 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:19.241861 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:19.242023 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:19.242072 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:19.242073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:19.242736 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:19.242737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:19.243305 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:19.243308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:19.243670 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:19.243671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:19.243717 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:19.249380 (+ 5663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:19.249387 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:19.251402 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
W20260928 22:54:19.251750 23542 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:19.241572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:43987 (local address 127.14.14.190:35131)
0928 22:54:19.241741 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:19.241743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:19.241750 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:54:19.241771 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:19.241772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:19.241782 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:19.241840 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:19.242146 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:19.242645 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:19.243399 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:19.243596 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:19.243651 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:19.251672 (+ 8021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:43987: BlockingRecv error: recv got EOF from 127.14.14.129:43987 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
Sep 28 22:54:19 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:19 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:19.367758 23543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:19.363264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:58861)
0928 22:54:19.363390 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:19.363400 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:19.363455 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:19.363583 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:19.363584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:19.363751 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:19.363798 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:19.363799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:19.364428 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:19.364429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:19.364960 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:19.364962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:19.365347 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:19.365348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:19.365389 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:19.365862 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:19.365867 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:19.367678 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
W20260928 22:54:19.367928 23542 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:19.363336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:58861 (local address 127.14.14.190:35131)
0928 22:54:19.363444 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:19.363446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:19.363455 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:54:19.363501 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:19.363502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:19.363514 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:19.363568 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:19.363870 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:19.364365 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:19.365087 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:19.365276 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:19.365330 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:19.367843 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:58861: BlockingRecv error: recv got EOF from 127.14.14.130:58861 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 22:54:19 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:19 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:19.498327 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:19.493007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:42965)
0928 22:54:19.493223 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:19.493234 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:19.493326 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:19.493479 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:19.493480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:19.493719 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:19.493811 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:19.493813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:19.494607 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:19.494609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:19.495269 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:19.495272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:19.495652 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:19.495653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:19.495708 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:19.496274 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:19.496278 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:19.498227 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":110,"threads_started":1}
W20260928 22:54:19.498383 23542 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:19.493121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:42965 (local address 127.14.14.190:35131)
0928 22:54:19.493200 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:19.493202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:19.493307 (+ 105us) server_negotiation.cc:407] Connection header received
0928 22:54:19.493383 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:19.493384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:19.493398 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:19.493474 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:19.493889 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:19.494524 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:19.495358 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:19.495573 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:19.495651 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:19.498313 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:42965: BlockingRecv error: recv got EOF from 127.14.14.131:42965 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 22:54:20 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:20 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:20.257264 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:20.252060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:47645)
0928 22:54:20.252202 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:20.252225 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:20.252275 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:20.252591 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:20.252592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:20.252761 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:20.252808 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:20.252810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:20.253520 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:20.253521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:20.254281 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:20.254284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:20.254817 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:20.254818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:20.254862 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:20.255336 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:20.255341 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:20.257175 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
W20260928 22:54:20.257330 23546 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:20.252211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:47645 (local address 127.14.14.190:35131)
0928 22:54:20.252458 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:20.252460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:20.252471 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:54:20.252495 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:20.252496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:20.252506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:20.252576 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:20.252882 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:20.253446 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:20.254404 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:20.254745 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:20.254806 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:20.257278 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:47645: BlockingRecv error: recv got EOF from 127.14.14.129:47645 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
Sep 28 22:54:20 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:20 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:20.373772 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:20.368275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:47441)
0928 22:54:20.368412 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:20.368425 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:20.368485 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:20.368667 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:20.368668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:20.368839 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:20.368893 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:20.368894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:20.369833 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:20.369834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:20.370385 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:20.370387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:20.370832 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:20.370834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:20.370889 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:20.371437 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:20.371443 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:20.373695 (+ 2252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
W20260928 22:54:20.373869 23546 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:20.368419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:47441 (local address 127.14.14.190:35131)
0928 22:54:20.368496 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:20.368498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:20.368508 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:54:20.368546 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:20.368546 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:20.368558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:20.368627 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:20.368979 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:20.369731 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:20.370494 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:20.370718 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:20.370786 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:20.373794 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:47441: BlockingRecv error: recv got EOF from 127.14.14.130:47441 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 22:54:20 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:20 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:20.503886 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:20.498852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:47743)
0928 22:54:20.499119 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:20.499129 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:20.499258 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:20.499410 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:20.499411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:20.499591 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:20.499640 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:20.499642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:20.500493 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:20.500495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:20.501040 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:20.501042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:20.501435 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:20.501436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:20.501496 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:20.501963 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:20.501967 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:20.503796 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":205,"threads_started":1}
W20260928 22:54:20.503959 23546 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:20.498932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:47743 (local address 127.14.14.190:35131)
0928 22:54:20.499021 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:20.499023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:20.499201 (+ 178us) server_negotiation.cc:407] Connection header received
0928 22:54:20.499334 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:20.499335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:20.499352 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:20.499406 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:20.499733 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:20.500422 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:20.501133 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:20.501348 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:20.501425 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:20.503892 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:47743: BlockingRecv error: recv got EOF from 127.14.14.131:47743 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 22:54:21 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:21 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:21.263245 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:21.257950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:34465)
0928 22:54:21.258100 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:21.258113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:21.258172 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:21.258372 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:21.258373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:21.258568 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:21.258640 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:21.258642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:21.259448 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:21.259449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:21.259987 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:21.259988 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:21.260457 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:21.260458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:21.260508 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:21.261196 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:21.261201 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:21.263186 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
W20260928 22:54:21.263521 23550 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:21.258077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:34465 (local address 127.14.14.190:35131)
0928 22:54:21.258208 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:21.258210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:21.258219 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:54:21.258243 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:21.258244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:21.258256 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:21.258338 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:21.258787 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:21.259370 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:21.260156 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:21.260362 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:21.260419 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:21.263432 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:34465: BlockingRecv error: recv got EOF from 127.14.14.129:34465 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
Sep 28 22:54:21 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:21 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:21.379397 23551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:21.374301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:46885)
0928 22:54:21.374461 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:21.374477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:21.374535 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:21.374680 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:21.374681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:21.374846 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:21.374900 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:21.374902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:21.375552 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:21.375553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:21.376080 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:21.376082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:21.376432 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:21.376433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:21.376473 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:21.376959 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:21.376963 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:21.379305 (+ 2342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
W20260928 22:54:21.379451 23550 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:21.374404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:46885 (local address 127.14.14.190:35131)
0928 22:54:21.374467 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:21.374469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:21.374539 (+ 70us) server_negotiation.cc:407] Connection header received
0928 22:54:21.374582 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:21.374583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:21.374594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:21.374649 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:21.374968 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:21.375471 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:21.376170 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:21.376364 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:21.376415 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:21.379390 (+ 2975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:46885: BlockingRecv error: recv got EOF from 127.14.14.130:46885 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 22:54:21 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:21 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:21.509044 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:21.504399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:41885)
0928 22:54:21.504533 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:21.504548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:21.504603 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:21.504741 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:21.504742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:21.504907 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:21.504952 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:21.504954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:21.505670 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:21.505672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:21.506196 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:21.506198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:21.506513 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:21.506513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:21.506552 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:21.506986 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:21.506990 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:21.508972 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
W20260928 22:54:21.509209 23550 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:21.504516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:41885 (local address 127.14.14.190:35131)
0928 22:54:21.504561 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:21.504567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:21.504589 (+ 22us) server_negotiation.cc:407] Connection header received
0928 22:54:21.504659 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:21.504659 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:21.504671 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:21.504725 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:21.505020 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:21.505518 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:21.506272 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:21.506451 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:21.506501 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:21.509127 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:41885: BlockingRecv error: recv got EOF from 127.14.14.131:41885 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 22:54:22 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:22 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:22.269233 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:22.263788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:37991)
0928 22:54:22.263910 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:22.263927 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:22.263989 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:22.264142 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:22.264143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:22.264305 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:22.264353 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:22.264355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:22.265127 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:22.265128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:22.266098 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:22.266102 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:22.266492 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:22.266493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:22.266540 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:22.267012 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:22.267016 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:22.269112 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
W20260928 22:54:22.269328 23554 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:22.263863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:37991 (local address 127.14.14.190:35131)
0928 22:54:22.264000 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:22.264002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:22.264016 (+ 14us) server_negotiation.cc:407] Connection header received
0928 22:54:22.264039 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:22.264040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:22.264050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:22.264116 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:22.264442 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:22.264958 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:22.266185 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:22.266413 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:22.266469 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:22.269233 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:37991: BlockingRecv error: recv got EOF from 127.14.14.129:37991 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
Sep 28 22:54:22 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:22 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:22.384768 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:22.379919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:60701)
0928 22:54:22.380155 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:22.380166 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:22.380232 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:22.380366 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:22.380367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:22.380531 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:22.380578 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:22.380579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:22.381286 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:22.381287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:22.381854 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:22.381856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:22.382227 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:22.382228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:22.382275 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:22.382730 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:22.382734 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:22.384708 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":181,"threads_started":1}
W20260928 22:54:22.384954 23554 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:22.380022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:60701 (local address 127.14.14.190:35131)
0928 22:54:22.380094 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:22.380096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:22.380236 (+ 140us) server_negotiation.cc:407] Connection header received
0928 22:54:22.380273 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:22.380273 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:22.380284 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:22.380336 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:22.380641 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:22.381185 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:22.381927 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:22.382139 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:22.382192 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:22.384838 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:60701: BlockingRecv error: recv got EOF from 127.14.14.130:60701 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 22:54:22 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:22 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:22.514673 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:22.509704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:36657)
0928 22:54:22.509975 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:22.509989 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:22.510089 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:22.510262 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:22.510263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:22.510450 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:22.510499 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:22.510500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:22.511205 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:22.511206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:22.511768 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:22.511770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:22.512151 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:22.512152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:22.512211 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:22.512731 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:22.512735 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:22.514575 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":77,"threads_started":1}
W20260928 22:54:22.514714 23554 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:22.509895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:36657 (local address 127.14.14.190:35131)
0928 22:54:22.509980 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:22.509982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:22.510074 (+ 92us) server_negotiation.cc:407] Connection header received
0928 22:54:22.510150 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:22.510151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:22.510167 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:22.510255 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:22.510586 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:22.511110 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:22.511865 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:22.512061 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:22.512149 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:22.514653 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:36657: BlockingRecv error: recv got EOF from 127.14.14.131:36657 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 22:54:23 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:23 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:23.274326 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:23.269851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:40373)
0928 22:54:23.269987 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:23.270002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:23.270070 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:23.270231 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:23.270231 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:23.270400 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:23.270447 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:23.270448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:23.271077 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:23.271077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:23.271609 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:23.271611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:23.271942 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:23.271943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:23.271981 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:23.272417 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:23.272422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:23.274259 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":56,"threads_started":1}
W20260928 22:54:23.274470 23558 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:23.269963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:40373 (local address 127.14.14.190:35131)
0928 22:54:23.270102 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:23.270105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:23.270113 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:54:23.270135 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:23.270136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:23.270146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:23.270202 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:23.270516 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:23.271009 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:23.271693 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:23.271876 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:23.271927 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:23.274379 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:40373: BlockingRecv error: recv got EOF from 127.14.14.129:40373 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
Sep 28 22:54:23 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:23 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:23.390475 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:23.385377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:49681)
0928 22:54:23.385515 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:23.385533 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:23.385664 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:23.385826 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:23.385826 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:23.385998 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:23.386045 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:23.386047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:23.386801 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:23.386802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:23.387377 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:23.387380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:23.387785 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:23.387786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:23.387846 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:23.388366 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:23.388371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:23.390368 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
W20260928 22:54:23.390532 23558 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:23.385448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:49681 (local address 127.14.14.190:35131)
0928 22:54:23.385537 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:23.385539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:23.385624 (+ 85us) server_negotiation.cc:407] Connection header received
0928 22:54:23.385736 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:23.385737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:23.385756 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:23.385812 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:23.386110 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:23.386683 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:23.387486 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:23.387684 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:23.387780 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:23.390465 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:49681: BlockingRecv error: recv got EOF from 127.14.14.130:49681 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 22:54:23 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:23 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:23.520454 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:23.515252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:32979)
0928 22:54:23.515455 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:23.515468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:23.515569 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:23.515769 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:23.515770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:23.515955 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:23.516003 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:23.516004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:23.516698 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:23.516699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:23.517321 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:23.517324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:23.517766 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:23.517768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:23.517842 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:23.518456 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:23.518461 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:23.520384 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":106,"threads_started":1}
W20260928 22:54:23.520684 23558 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:23.515383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:32979 (local address 127.14.14.190:35131)
0928 22:54:23.515459 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:23.515461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:23.515548 (+ 87us) server_negotiation.cc:407] Connection header received
0928 22:54:23.515646 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:23.515647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:23.515666 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:23.515728 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:23.516088 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:23.516605 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:23.517417 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:23.517683 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:23.517757 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:23.520557 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:32979: BlockingRecv error: recv got EOF from 127.14.14.131:32979 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 22:54:24 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:24 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:24.286146 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:24.274881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:52851)
0928 22:54:24.275004 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:24.275019 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:24.275070 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:24.275246 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:24.275247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:24.275414 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:24.275459 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:24.275460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:24.276232 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:24.276233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:24.276791 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:24.276792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:24.277239 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:24.277240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:24.277292 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:24.284018 (+ 6726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:24.284023 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:24.286085 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
W20260928 22:54:24.286288 23562 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:24.274946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:52851 (local address 127.14.14.190:35131)
0928 22:54:24.275126 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:24.275128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:24.275136 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:54:24.275157 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:24.275157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:24.275168 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:24.275232 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:24.275550 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:24.276147 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:24.276881 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:24.277146 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:24.277205 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:24.286198 (+ 8993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:52851: BlockingRecv error: recv got EOF from 127.14.14.129:52851 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":109,"threads_started":1}
Sep 28 22:54:24 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:24 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:24.396765 23563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:24.391213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:46047)
0928 22:54:24.391402 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:24.391416 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:24.391487 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:24.391652 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:24.391652 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:24.391824 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:24.391872 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:24.391873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:24.392748 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:24.392750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:24.393426 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:24.393429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:24.393797 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:24.393798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:24.393842 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:24.394500 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:24.394504 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:24.396675 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
W20260928 22:54:24.396901 23562 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:24.391358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:46047 (local address 127.14.14.190:35131)
0928 22:54:24.391440 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:24.391442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:24.391470 (+ 28us) server_negotiation.cc:407] Connection header received
0928 22:54:24.391554 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:24.391555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:24.391565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:24.391627 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:24.391937 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:24.392626 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:24.393497 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:24.393722 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:24.393775 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:24.396833 (+ 3058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:46047: BlockingRecv error: recv got EOF from 127.14.14.130:46047 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 22:54:24 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:24 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:24.525501 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:24.520907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:39135)
0928 22:54:24.521083 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:24.521095 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:24.521214 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:24.521342 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:24.521342 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:24.521514 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:24.521562 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:24.521564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:24.522233 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:24.522233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:24.522764 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:24.522765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:24.523138 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:24.523139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:24.523189 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:24.523653 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:24.523658 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:24.525441 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":120,"threads_started":1}
W20260928 22:54:24.525621 23562 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:24.520964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:39135 (local address 127.14.14.190:35131)
0928 22:54:24.521046 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:24.521048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:24.521214 (+ 166us) server_negotiation.cc:407] Connection header received
0928 22:54:24.521260 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:24.521261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:24.521273 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:24.521321 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:24.521672 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:24.522159 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:24.522840 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:24.523018 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:24.523075 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:24.525530 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:39135: BlockingRecv error: recv got EOF from 127.14.14.131:39135 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 22:54:25 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:25 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:25.291559 23565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:25.286803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:56795)
0928 22:54:25.286947 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:25.286958 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:25.287006 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:25.287209 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:25.287210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:25.287376 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:25.287422 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:25.287423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:25.288059 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:25.288059 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:25.288655 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:25.288657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:25.288989 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:25.288989 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:25.289028 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:25.289521 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:25.289525 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:25.291473 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
W20260928 22:54:25.291674 23566 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:25.286934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:56795 (local address 127.14.14.190:35131)
0928 22:54:25.287063 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:25.287065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:25.287074 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:54:25.287103 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:25.287104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:25.287120 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:25.287187 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:25.287495 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:25.287991 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:25.288732 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:25.288923 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:25.288974 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:25.291566 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:56795: BlockingRecv error: recv got EOF from 127.14.14.129:56795 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
Sep 28 22:54:25 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:25 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:25.403102 23567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:25.397358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:51075)
0928 22:54:25.397522 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:25.397536 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:25.397633 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:25.397831 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:25.397832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:25.398024 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:25.398069 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:25.398071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:25.398852 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:25.398854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:25.399425 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:25.399426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:25.399982 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:25.399982 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:25.400050 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:25.400612 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:25.400618 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:25.402994 (+ 2376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
W20260928 22:54:25.403160 23566 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:25.397493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:51075 (local address 127.14.14.190:35131)
0928 22:54:25.397549 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:25.397551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:25.397624 (+ 73us) server_negotiation.cc:407] Connection header received
0928 22:54:25.397732 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:25.397737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:25.397751 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:25.397826 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:25.398167 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:25.398760 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:25.399549 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:25.399899 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:25.399981 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:25.403090 (+ 3109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:51075: BlockingRecv error: recv got EOF from 127.14.14.130:51075 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 22:54:25 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:25 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:25.530623 23568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:25.526001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:53325)
0928 22:54:25.526211 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:25.526223 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:25.526297 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:25.526426 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:25.526427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:25.526599 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:25.526648 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:25.526650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:25.527290 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:25.527291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:25.527848 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:25.527850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:25.528161 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:25.528162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:25.528204 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:25.528644 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:25.528648 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:25.530547 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":120,"threads_started":1}
W20260928 22:54:25.530706 23566 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:25.526160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:53325 (local address 127.14.14.190:35131)
0928 22:54:25.526229 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:25.526231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:25.526295 (+ 64us) server_negotiation.cc:407] Connection header received
0928 22:54:25.526350 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:25.526351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:25.526363 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:25.526402 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:25.526710 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:25.527199 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:25.527921 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:25.528097 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:25.528144 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:25.530660 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:53325: BlockingRecv error: recv got EOF from 127.14.14.131:53325 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 22:54:26 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:26 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:26.296737 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:26.292048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:57817)
0928 22:54:26.292234 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:26.292282 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:26.292369 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:26.292507 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:26.292508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:26.292669 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:26.292716 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:26.292718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:26.293350 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:26.293350 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:26.293964 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:26.293967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:26.294390 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:26.294391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:26.294439 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:26.294927 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:26.294931 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:26.296679 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":79,"threads_started":1}
W20260928 22:54:26.296877 23570 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:26.292178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:57817 (local address 127.14.14.190:35131)
0928 22:54:26.292325 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:26.292327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:26.292352 (+ 25us) server_negotiation.cc:407] Connection header received
0928 22:54:26.292421 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:26.292422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:26.292433 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:26.292481 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:26.292781 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:26.293279 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:26.294041 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:26.294293 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:26.294372 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:26.296773 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:57817: BlockingRecv error: recv got EOF from 127.14.14.129:57817 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
Sep 28 22:54:26 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:26 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:26.409495 23571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:26.403871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:60775)
0928 22:54:26.404029 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:26.404038 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:26.404124 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:26.404342 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:26.404343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:26.404589 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:26.404651 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:26.404653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:26.405444 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:26.405446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:26.406420 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:26.406423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:26.406804 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:26.406805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:26.406864 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:26.407476 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:26.407483 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:26.409436 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
W20260928 22:54:26.409667 23570 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:26.403922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:60775 (local address 127.14.14.190:35131)
0928 22:54:26.403981 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:26.403983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:26.404132 (+ 149us) server_negotiation.cc:407] Connection header received
0928 22:54:26.404204 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:26.404205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:26.404223 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:26.404318 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:26.404723 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:26.405350 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:26.406498 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:26.406740 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:26.406806 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:26.409550 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:60775: BlockingRecv error: recv got EOF from 127.14.14.130:60775 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 22:54:26 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:26 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.131@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:26.535521 23572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:26.531146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.131:47929)
0928 22:54:26.531295 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:26.531306 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:26.531359 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:26.531470 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:26.531470 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:26.531640 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:26.531688 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:26.531690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:26.532348 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:26.532349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:26.532872 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:26.532873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:26.533181 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:26.533182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:26.533220 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:26.533687 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:26.533693 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:26.535463 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
W20260928 22:54:26.535667 23570 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:26.531293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:47929 (local address 127.14.14.190:35131)
0928 22:54:26.531366 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:26.531368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:26.531377 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:54:26.531402 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:26.531403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:26.531415 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:26.531464 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:26.531753 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:26.532242 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:26.532944 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:26.533119 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:26.533168 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:26.535583 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.131:47929: BlockingRecv error: recv got EOF from 127.14.14.131:47929 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 22:54:27 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:27 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.129@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:27.302690 23573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:27.297350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.129:45763)
0928 22:54:27.297498 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:27.297508 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:27.297557 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:27.297799 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:27.297799 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:27.297963 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:27.298014 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:27.298016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.299033 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.299034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:27.299620 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:27.299623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.299971 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.299972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:27.300013 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:27.300488 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:27.300492 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:27.302604 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
W20260928 22:54:27.302796 23574 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:27.297424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:45763 (local address 127.14.14.190:35131)
0928 22:54:27.297545 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:27.297547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:27.297558 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:54:27.297674 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:27.297675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:27.297688 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:27.297777 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:27.298086 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.298953 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.299702 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.299900 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.299958 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:27.302699 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.129:45763: BlockingRecv error: recv got EOF from 127.14.14.129:45763 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
Sep 28 22:54:27 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:54:27 dist-test-slave-kzfr krb5kdc[22516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.14.130@KRBTEST.COM for oryx/127.14.14.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:54:27.415149 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:27.410136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:35131 (local address 127.14.14.130:58581)
0928 22:54:27.410265 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:27.410278 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:27.410360 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:27.410523 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:27.410523 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:27.410698 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:27.410749 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:27.410751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.411522 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.411523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:27.412405 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:27.412407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.412721 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.412721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:27.412760 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:27.413206 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:27.413211 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:27.415090 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.14.190:35131: Server oryx/127.14.14.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
W20260928 22:54:27.415309 23574 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:27.410210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:58581 (local address 127.14.14.190:35131)
0928 22:54:27.410260 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:27.410263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:27.410336 (+ 73us) server_negotiation.cc:407] Connection header received
0928 22:54:27.410425 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:27.410427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:27.410439 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:27.410496 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:27.410849 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.411387 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.412459 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.412656 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.412710 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:27.415221 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.14.130:58581: BlockingRecv error: recv got EOF from 127.14.14.130:58581 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:54:27.444108 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 23099
I20260928 22:54:27.448593 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 23251
I20260928 22:54:27.452842 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 23383
I20260928 22:54:27.456911 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 23029
2026-09-28T22:54:27Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17431 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T22:54:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:54:27Z Disabled control of system clock
I20260928 22:54:27.484773 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:32861
--webserver_interface=127.14.14.190
--webserver_port=0
--builtin_ntp_servers=127.14.14.148:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:32861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 22:54:27.566437 23579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:27.566655 23579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:27.566677 23579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:27.567973 23579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:54:27.568120 23579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:27.568166 23579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:27.568210 23579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:54:27.568239 23579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:54:27.569717 23579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:32861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:32861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:27.569969 23579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:27.570312 23579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:54:27.576201 23579 server_base.cc:1034] running on GCE node
W20260928 22:54:27.576221 23585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:27.576164 23587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:27.576141 23584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:27.576623 23579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:27.576944 23579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:27.578167 23579 hybrid_clock.cc:648] HybridClock initialized: now 1790636067578159 us; error 45 us; skew 500 ppm
I20260928 22:54:27.579470 23579 webserver.cc:460] Webserver started at http://127.14.14.190:34665/ using document root <none> and password file <none>
I20260928 22:54:27.579732 23579 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:27.579807 23579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:27.579962 23579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:27.581106 23579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "c8997897a30d4703bc926633f1100bd2"
format_stamp: "Formatted at 2026-09-28 22:54:27 on dist-test-slave-kzfr"
server_key: "db5ef435e002f8afd238603ba78953f2"
server_key_iv: "a869762182031bf0ba895c35749d8a20"
server_key_version: "encryptionkey@0"
I20260928 22:54:27.581485 23579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "c8997897a30d4703bc926633f1100bd2"
format_stamp: "Formatted at 2026-09-28 22:54:27 on dist-test-slave-kzfr"
server_key: "db5ef435e002f8afd238603ba78953f2"
server_key_iv: "a869762182031bf0ba895c35749d8a20"
server_key_version: "encryptionkey@0"
I20260928 22:54:27.582805 23579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:54:27.583581 23593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:27.583767 23579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:54:27.583866 23579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "c8997897a30d4703bc926633f1100bd2"
format_stamp: "Formatted at 2026-09-28 22:54:27 on dist-test-slave-kzfr"
server_key: "db5ef435e002f8afd238603ba78953f2"
server_key_iv: "a869762182031bf0ba895c35749d8a20"
server_key_version: "encryptionkey@0"
I20260928 22:54:27.583950 23579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:27.595664 23579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:27.595970 23579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:27.596132 23579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:27.600420 23579 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:32861
I20260928 22:54:27.600454 23645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:32861 every 8 connection(s)
I20260928 22:54:27.600792 23579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:54:27.601518 23646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:27.604094 23646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2: Bootstrap starting.
I20260928 22:54:27.604715 23646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:27.605261 23646 log.cc:824] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:27.606081 23646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2: No bootstrap required, opened a new log
I20260928 22:54:27.607751 23646 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8997897a30d4703bc926633f1100bd2" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 32861 } }
I20260928 22:54:27.607882 23646 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:27.607905 23646 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8997897a30d4703bc926633f1100bd2, State: Initialized, Role: FOLLOWER
I20260928 22:54:27.608065 23646 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8997897a30d4703bc926633f1100bd2" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 32861 } }
I20260928 22:54:27.608142 23646 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:54:27.608165 23646 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:54:27.608237 23646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:27.608752 23646 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8997897a30d4703bc926633f1100bd2" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 32861 } }
I20260928 22:54:27.608862 23646 leader_election.cc:302] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8997897a30d4703bc926633f1100bd2; no voters:
I20260928 22:54:27.609057 23646 leader_election.cc:288] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:54:27.609089 23649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:54:27.609077 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 23579
I20260928 22:54:27.609212 23649 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [term 1 LEADER]: Becoming Leader. State: Replica: c8997897a30d4703bc926633f1100bd2, State: Running, Role: LEADER
I20260928 22:54:27.609267 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:54:27.609292 23649 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8997897a30d4703bc926633f1100bd2" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 32861 } }
I20260928 22:54:27.609325 23646 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:54:27.609472 14394 external_mini_cluster.cc:1428] Setting key f174de1fca28d285f8124a118da379d8
I20260928 22:54:27.609702 23649 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8997897a30d4703bc926633f1100bd2. Latest consensus state: current_term: 1 leader_uuid: "c8997897a30d4703bc926633f1100bd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8997897a30d4703bc926633f1100bd2" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 32861 } } }
I20260928 22:54:27.610106 23650 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8997897a30d4703bc926633f1100bd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8997897a30d4703bc926633f1100bd2" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 32861 } } }
I20260928 22:54:27.610190 23649 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:27.610433 23650 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2 [sys.catalog]: This master's current role is: LEADER
W20260928 22:54:27.611189 23667 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:54:27.611296 23667 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:54:27.611426 23662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:54:27.611632 23662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:54:27.613023 23662 catalog_manager.cc:1347] Generated new cluster ID: 10f8c31a8a1b4a218167cfa2fefa560f
I20260928 22:54:27.613073 23662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:54:27.616281 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:27.610181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.14.14.190:32861)
0928 22:54:27.610873 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:27.610883 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:27.610919 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:54:27.611037 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:27.611044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:27.611115 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:27.611278 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:27.611538 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.612879 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.614005 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.614600 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.614709 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:27.615467 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:27.615471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:27.615472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:54:27.615680 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:27.615720 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:54:27.615733 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:27.615797 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:27.615896 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 22:54:27.616062 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":90,"threads_started":1}
I20260928 22:54:27.629194 23662 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:54:27.629778 23662 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:54:27.637261 23662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8997897a30d4703bc926633f1100bd2: Generated new TSK 0
I20260928 22:54:27.637449 23662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:54:27.644475 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:32861
--builtin_ntp_servers=127.14.14.148:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 22:54:27.727367 23670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:27.727617 23670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:27.727655 23670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:27.728896 23670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:27.728957 23670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:27.728999 23670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:54:27.730437 23670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:32861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:27.730716 23670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:27.731019 23670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:54:27.736310 23670 server_base.cc:1034] running on GCE node
W20260928 22:54:27.736285 23678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:27.736294 23675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:27.736507 23676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:27.736780 23670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:27.737046 23670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:27.738220 23670 hybrid_clock.cc:648] HybridClock initialized: now 1790636067738244 us; error 67 us; skew 500 ppm
I20260928 22:54:27.739576 23670 webserver.cc:460] Webserver started at http://127.14.14.129:43957/ using document root <none> and password file <none>
I20260928 22:54:27.739856 23670 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:27.739933 23670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:27.740092 23670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:27.741233 23670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "c8334892debc4d1ea47af09d05d766aa"
format_stamp: "Formatted at 2026-09-28 22:54:27 on dist-test-slave-kzfr"
server_key: "59a8ee72dbe254512df677e3cc1f3929"
server_key_iv: "8342cfdad792354ea62420c13e1dfcf0"
server_key_version: "encryptionkey@0"
I20260928 22:54:27.741668 23670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "c8334892debc4d1ea47af09d05d766aa"
format_stamp: "Formatted at 2026-09-28 22:54:27 on dist-test-slave-kzfr"
server_key: "59a8ee72dbe254512df677e3cc1f3929"
server_key_iv: "8342cfdad792354ea62420c13e1dfcf0"
server_key_version: "encryptionkey@0"
I20260928 22:54:27.742980 23670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:54:27.743820 23684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:27.744086 23670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:54:27.744194 23670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "c8334892debc4d1ea47af09d05d766aa"
format_stamp: "Formatted at 2026-09-28 22:54:27 on dist-test-slave-kzfr"
server_key: "59a8ee72dbe254512df677e3cc1f3929"
server_key_iv: "8342cfdad792354ea62420c13e1dfcf0"
server_key_version: "encryptionkey@0"
I20260928 22:54:27.744285 23670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:27.751396 23670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:27.751665 23670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:27.751816 23670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:27.752020 23670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:27.752339 23670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:27.752398 23670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:27.752437 23670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:27.752465 23670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:27.758199 23670 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:38049
I20260928 22:54:27.758255 23797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:38049 every 8 connection(s)
I20260928 22:54:27.758556 23670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:54:27.759243 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 23670
I20260928 22:54:27.759348 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:54:27.759557 14394 external_mini_cluster.cc:1428] Setting key 7382c458f1c87e7b07dc5dc9e6351303
I20260928 22:54:27.760648 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:32861
--builtin_ntp_servers=127.14.14.148:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 22:54:27.764438 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:27.759273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:35499 (local address 127.14.14.190:32861)
0928 22:54:27.759343 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:27.759345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:27.759570 (+ 225us) server_negotiation.cc:407] Connection header received
0928 22:54:27.760220 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:27.760221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:27.760234 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:27.760295 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:27.761223 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.761783 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.762850 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.763035 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.763090 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:27.764033 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:27.764035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:27.764035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:54:27.764040 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:27.764059 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:54:27.764063 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:27.764201 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:27.764286 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 22:54:27.764317 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:54:27.764528 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:27.759232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:32861 (local address 127.14.14.129:35499)
0928 22:54:27.759435 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:27.759473 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:27.760080 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:27.760369 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:27.760374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:27.760385 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:27.761119 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:27.761126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.761880 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.761881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:27.762759 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:27.762761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.763133 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.763134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:27.763307 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:27.763872 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:54:27.763872 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:27.763909 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:54:27.763937 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:27.764198 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:27.764203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:27.764205 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:54:27.764266 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 22:54:27.764346 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260928 22:54:27.765064 23798 heartbeater.cc:347] Connected to a master server at 127.14.14.190:32861
I20260928 22:54:27.765157 23798 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:27.765355 23798 heartbeater.cc:511] Master 127.14.14.190:32861 requested a full tablet report, sending...
I20260928 22:54:27.765931 23610 ts_manager.cc:194] Registered new tserver with Master: c8334892debc4d1ea47af09d05d766aa (127.14.14.129:38049)
I20260928 22:54:27.766929 23610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.14.129:35499
W20260928 22:54:27.847628 23801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:27.847844 23801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:27.847865 23801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:27.849027 23801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:27.849069 23801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:27.849094 23801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:54:27.850382 23801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:32861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:27.850581 23801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:27.850835 23801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:27.855902 23806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:27.855913 23807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:27.855918 23809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:27.856097 23801 server_base.cc:1034] running on GCE node
I20260928 22:54:27.856462 23801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:27.856714 23801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:27.857858 23801 hybrid_clock.cc:648] HybridClock initialized: now 1790636067857846 us; error 30 us; skew 500 ppm
I20260928 22:54:27.859282 23801 webserver.cc:460] Webserver started at http://127.14.14.130:41887/ using document root <none> and password file <none>
I20260928 22:54:27.859536 23801 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:27.859608 23801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:27.859784 23801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:27.860980 23801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "e7939529300e4538b997fa5078cbacfa"
format_stamp: "Formatted at 2026-09-28 22:54:27 on dist-test-slave-kzfr"
server_key: "f0eff7a0492edc30e58f44e1ea6059ec"
server_key_iv: "c32cbc37e13aec5b52adbbfa00188356"
server_key_version: "encryptionkey@0"
I20260928 22:54:27.861315 23801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "e7939529300e4538b997fa5078cbacfa"
format_stamp: "Formatted at 2026-09-28 22:54:27 on dist-test-slave-kzfr"
server_key: "f0eff7a0492edc30e58f44e1ea6059ec"
server_key_iv: "c32cbc37e13aec5b52adbbfa00188356"
server_key_version: "encryptionkey@0"
I20260928 22:54:27.862625 23801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:54:27.863327 23815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:27.863482 23801 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:27.863550 23801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "e7939529300e4538b997fa5078cbacfa"
format_stamp: "Formatted at 2026-09-28 22:54:27 on dist-test-slave-kzfr"
server_key: "f0eff7a0492edc30e58f44e1ea6059ec"
server_key_iv: "c32cbc37e13aec5b52adbbfa00188356"
server_key_version: "encryptionkey@0"
I20260928 22:54:27.863602 23801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:27.888998 23801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:27.889277 23801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:27.889381 23801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:27.889561 23801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:27.889964 23801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:27.889997 23801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:27.890019 23801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:27.890033 23801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:27.895839 23801 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:43021
I20260928 22:54:27.895886 23928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:43021 every 8 connection(s)
I20260928 22:54:27.896222 23801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:54:27.897490 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 23801
I20260928 22:54:27.897603 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:54:27.897814 14394 external_mini_cluster.cc:1428] Setting key dac5dd8a6304f61acfa56ecbc04a73c6
I20260928 22:54:27.898916 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:32861
--builtin_ntp_servers=127.14.14.148:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 22:54:27.902503 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:27.896969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:50487 (local address 127.14.14.190:32861)
0928 22:54:27.897016 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:27.897018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:27.897289 (+ 271us) server_negotiation.cc:407] Connection header received
0928 22:54:27.898074 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:27.898076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:27.898089 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:27.898224 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:27.899187 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.899657 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.900866 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.901195 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.901275 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:27.902146 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:27.902147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:27.902148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:54:27.902153 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:27.902171 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:54:27.902175 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:27.902240 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:27.902355 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 22:54:27.902385 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:54:27.902604 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:27.896972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:32861 (local address 127.14.14.130:50487)
0928 22:54:27.897160 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:27.897194 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:27.897960 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:27.898326 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:27.898330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:27.898338 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:27.899067 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:27.899074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.899764 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.899765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:27.900741 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:27.900743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:27.901288 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:27.901289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:27.901404 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:27.901995 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:54:27.901995 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:27.902031 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:54:27.902054 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:27.902258 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:27.902264 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:27.902267 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:54:27.902340 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:54:27.902420 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260928 22:54:27.903105 23929 heartbeater.cc:347] Connected to a master server at 127.14.14.190:32861
I20260928 22:54:27.903187 23929 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:27.903378 23929 heartbeater.cc:511] Master 127.14.14.190:32861 requested a full tablet report, sending...
I20260928 22:54:27.903771 23610 ts_manager.cc:194] Registered new tserver with Master: e7939529300e4538b997fa5078cbacfa (127.14.14.130:43021)
I20260928 22:54:27.904315 23610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.14.130:50487
W20260928 22:54:27.981623 23932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:27.981849 23932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:27.981870 23932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:27.982954 23932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:27.982988 23932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:27.983012 23932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:54:27.984258 23932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:32861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:27.984443 23932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:27.984674 23932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:27.990060 23937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:27.990051 23940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:27.990077 23938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:27.990236 23932 server_base.cc:1034] running on GCE node
I20260928 22:54:27.990566 23932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:27.990839 23932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:27.992012 23932 hybrid_clock.cc:648] HybridClock initialized: now 1790636067992000 us; error 46 us; skew 500 ppm
I20260928 22:54:27.993469 23932 webserver.cc:460] Webserver started at http://127.14.14.131:36397/ using document root <none> and password file <none>
I20260928 22:54:27.993815 23932 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:27.993914 23932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:27.994091 23932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:27.995258 23932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "cb80a2103e0b46d9aa780a7b7554eff2"
format_stamp: "Formatted at 2026-09-28 22:54:27 on dist-test-slave-kzfr"
server_key: "15780ca89bf27f1a4c5e5604d59a63c0"
server_key_iv: "3eea704048050a42c4d06f5beaeaecc0"
server_key_version: "encryptionkey@0"
I20260928 22:54:27.995662 23932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "cb80a2103e0b46d9aa780a7b7554eff2"
format_stamp: "Formatted at 2026-09-28 22:54:27 on dist-test-slave-kzfr"
server_key: "15780ca89bf27f1a4c5e5604d59a63c0"
server_key_iv: "3eea704048050a42c4d06f5beaeaecc0"
server_key_version: "encryptionkey@0"
I20260928 22:54:27.997085 23932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:54:27.997901 23946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:27.998068 23932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:54:27.998179 23932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "cb80a2103e0b46d9aa780a7b7554eff2"
format_stamp: "Formatted at 2026-09-28 22:54:27 on dist-test-slave-kzfr"
server_key: "15780ca89bf27f1a4c5e5604d59a63c0"
server_key_iv: "3eea704048050a42c4d06f5beaeaecc0"
server_key_version: "encryptionkey@0"
I20260928 22:54:27.998256 23932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:28.011518 23932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:28.011868 23932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:28.012033 23932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:28.012279 23932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:28.012624 23932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:28.012689 23932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:28.012732 23932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:28.012754 23932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:28.018519 23932 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:46241
I20260928 22:54:28.018632 24059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:46241 every 8 connection(s)
I20260928 22:54:28.018976 23932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:54:28.024235 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 23932
I20260928 22:54:28.024344 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:54:28.024587 14394 external_mini_cluster.cc:1428] Setting key 3f522682b1d8553066747c2effb049ea
I20260928 22:54:28.024773 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:28.019763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:58209 (local address 127.14.14.190:32861)
0928 22:54:28.019810 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:28.019812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:28.020122 (+ 310us) server_negotiation.cc:407] Connection header received
0928 22:54:28.020683 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:28.020685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:28.020696 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:28.020754 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:28.021605 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:28.022109 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:28.023128 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:28.023321 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:28.023432 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:28.024386 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:28.024387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:28.024388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:54:28.024394 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:28.024411 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:54:28.024418 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:28.024478 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:28.024614 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 22:54:28.024679 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:54:28.024951 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:28.019728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:32861 (local address 127.14.14.131:58209)
0928 22:54:28.019975 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:28.020020 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:28.020576 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:28.020834 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:28.020838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:28.020848 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:28.021459 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:28.021464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:28.022207 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:28.022208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:28.023007 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:28.023009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:28.023439 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:28.023440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:28.023581 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:28.024145 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:54:28.024146 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:28.024189 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:54:28.024222 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:28.024486 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:28.024492 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:28.024496 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:54:28.024577 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 22:54:28.024694 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":100,"threads_started":1}
I20260928 22:54:28.025700 24060 heartbeater.cc:347] Connected to a master server at 127.14.14.190:32861
I20260928 22:54:28.025800 24060 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:28.025982 24060 heartbeater.cc:511] Master 127.14.14.190:32861 requested a full tablet report, sending...
I20260928 22:54:28.026383 23609 ts_manager.cc:194] Registered new tserver with Master: cb80a2103e0b46d9aa780a7b7554eff2 (127.14.14.131:46241)
I20260928 22:54:28.026717 23609 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.14.131:58209
I20260928 22:54:28.036862 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 22:54:28.038951 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:28.038426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.14.14.190:32861)
0928 22:54:28.038494 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:28.038496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:28.038611 (+ 115us) server_negotiation.cc:407] Connection header received
0928 22:54:28.038651 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:28.038651 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:28.038663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:28.038721 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:28.038870 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50072: BlockingRecv error: recv got EOF from 127.0.0.1:50072 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:54:28.039037 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 23670
I20260928 22:54:28.044050 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 23801
I20260928 22:54:28.048455 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 23932
I20260928 22:54:28.052709 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 23579
2026-09-28T22:54:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (594 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T22:54:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:54:28Z Disabled control of system clock
I20260928 22:54:28.079836 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:39929
--webserver_interface=127.14.14.190
--webserver_port=0
--builtin_ntp_servers=127.14.14.148:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:39929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 22:54:28.163022 24071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:28.163237 24071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:28.163265 24071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:28.164489 24071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:54:28.164533 24071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:28.164547 24071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:28.164561 24071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:54:28.164572 24071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:54:28.165932 24071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:39929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:39929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:28.166209 24071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:28.166517 24071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:28.172125 24076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:28.172133 24079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:28.172147 24077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:28.172322 24071 server_base.cc:1034] running on GCE node
I20260928 22:54:28.172654 24071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:28.172986 24071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:28.174173 24071 hybrid_clock.cc:648] HybridClock initialized: now 1790636068174153 us; error 39 us; skew 500 ppm
I20260928 22:54:28.175493 24071 webserver.cc:460] Webserver started at http://127.14.14.190:45547/ using document root <none> and password file <none>
I20260928 22:54:28.175789 24071 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:28.175858 24071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:28.176008 24071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:28.177124 24071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "bd736b6ccdf04e34b6464351f3b4e097"
format_stamp: "Formatted at 2026-09-28 22:54:28 on dist-test-slave-kzfr"
server_key: "c46d99a38fef1512d6f70b7bedbe6f5c"
server_key_iv: "0217634385e84c93ac066d540bbc70d6"
server_key_version: "encryptionkey@0"
I20260928 22:54:28.177505 24071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "bd736b6ccdf04e34b6464351f3b4e097"
format_stamp: "Formatted at 2026-09-28 22:54:28 on dist-test-slave-kzfr"
server_key: "c46d99a38fef1512d6f70b7bedbe6f5c"
server_key_iv: "0217634385e84c93ac066d540bbc70d6"
server_key_version: "encryptionkey@0"
I20260928 22:54:28.178864 24071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:54:28.179659 24085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:28.179833 24071 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:54:28.179935 24071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "bd736b6ccdf04e34b6464351f3b4e097"
format_stamp: "Formatted at 2026-09-28 22:54:28 on dist-test-slave-kzfr"
server_key: "c46d99a38fef1512d6f70b7bedbe6f5c"
server_key_iv: "0217634385e84c93ac066d540bbc70d6"
server_key_version: "encryptionkey@0"
I20260928 22:54:28.180006 24071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:28.183555 24071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:28.183864 24071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:28.184005 24071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:28.187846 24071 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:39929
I20260928 22:54:28.187875 24137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:39929 every 8 connection(s)
I20260928 22:54:28.188302 24071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:54:28.189016 24138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:28.191761 24138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097: Bootstrap starting.
I20260928 22:54:28.192282 24138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:28.192528 24138 log.cc:824] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:28.193135 24138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097: No bootstrap required, opened a new log
I20260928 22:54:28.194160 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 24071
I20260928 22:54:28.194243 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:54:28.194446 14394 external_mini_cluster.cc:1428] Setting key ee47b389a5c53f38fcdd2151c7944576
I20260928 22:54:28.195468 24138 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd736b6ccdf04e34b6464351f3b4e097" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39929 } }
I20260928 22:54:28.195649 24138 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:28.195688 24138 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd736b6ccdf04e34b6464351f3b4e097, State: Initialized, Role: FOLLOWER
I20260928 22:54:28.195842 24138 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd736b6ccdf04e34b6464351f3b4e097" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39929 } }
I20260928 22:54:28.195938 24138 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:54:28.195969 24138 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:54:28.196038 24138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:28.196728 24138 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd736b6ccdf04e34b6464351f3b4e097" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39929 } }
I20260928 22:54:28.196831 24138 leader_election.cc:302] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd736b6ccdf04e34b6464351f3b4e097; no voters:
I20260928 22:54:28.196975 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:28.195038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56448 (local address 127.14.14.190:39929)
0928 22:54:28.195311 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:28.195320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:28.195355 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:54:28.195473 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:28.195480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:28.195525 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:28.195673 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:28.196461 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:28.196465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:28.196465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:54:28.196470 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:28.196519 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:54:28.196533 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:28.196613 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:28.196684 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 22:54:28.196743 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":139,"threads_started":1}
I20260928 22:54:28.196995 24138 leader_election.cc:288] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:54:28.197239 24138 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:54:28.197384 24143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:54:28.197536 24143 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [term 1 LEADER]: Becoming Leader. State: Replica: bd736b6ccdf04e34b6464351f3b4e097, State: Running, Role: LEADER
I20260928 22:54:28.197716 24143 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd736b6ccdf04e34b6464351f3b4e097" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39929 } }
I20260928 22:54:28.198156 24143 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd736b6ccdf04e34b6464351f3b4e097. Latest consensus state: current_term: 1 leader_uuid: "bd736b6ccdf04e34b6464351f3b4e097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd736b6ccdf04e34b6464351f3b4e097" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39929 } } }
I20260928 22:54:28.198298 24143 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:28.198207 24145 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd736b6ccdf04e34b6464351f3b4e097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd736b6ccdf04e34b6464351f3b4e097" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39929 } } }
I20260928 22:54:28.198899 24145 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097 [sys.catalog]: This master's current role is: LEADER
W20260928 22:54:28.199163 24158 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:54:28.199203 24158 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:54:28.199288 24156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:54:28.199477 24156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:54:28.200855 24156 catalog_manager.cc:1347] Generated new cluster ID: 8ed897c231744d4eb029f67321b03746
I20260928 22:54:28.200903 24156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:54:28.209648 24156 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:54:28.209806 24156 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:54:28.215531 24156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd736b6ccdf04e34b6464351f3b4e097: Generated new TSK 0
I20260928 22:54:28.215699 24156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:54:28.217696 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:39929
--builtin_ntp_servers=127.14.14.148:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 22:54:28.300786 24162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:28.300978 24162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:28.301000 24162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:28.302270 24162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:28.302306 24162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:28.302332 24162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:54:28.303645 24162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:39929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:28.303869 24162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:28.304169 24162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:28.310014 24167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:28.309990 24168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:28.310395 24162 server_base.cc:1034] running on GCE node
W20260928 22:54:28.309994 24170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:28.310663 24162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:28.310933 24162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:28.312103 24162 hybrid_clock.cc:648] HybridClock initialized: now 1790636068312089 us; error 30 us; skew 500 ppm
I20260928 22:54:28.313421 24162 webserver.cc:460] Webserver started at http://127.14.14.129:44757/ using document root <none> and password file <none>
I20260928 22:54:28.313769 24162 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:28.313841 24162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:28.313977 24162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:28.315177 24162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "076b4ed0fa8c48b1b7bfca8eed4819ae"
format_stamp: "Formatted at 2026-09-28 22:54:28 on dist-test-slave-kzfr"
server_key: "721f4088e259afe139ce043e1f1c8126"
server_key_iv: "a4077f59ef833935fd68c5102645979b"
server_key_version: "encryptionkey@0"
I20260928 22:54:28.315541 24162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "076b4ed0fa8c48b1b7bfca8eed4819ae"
format_stamp: "Formatted at 2026-09-28 22:54:28 on dist-test-slave-kzfr"
server_key: "721f4088e259afe139ce043e1f1c8126"
server_key_iv: "a4077f59ef833935fd68c5102645979b"
server_key_version: "encryptionkey@0"
I20260928 22:54:28.316979 24162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:54:28.317808 24176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:28.317981 24162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:28.318064 24162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "076b4ed0fa8c48b1b7bfca8eed4819ae"
format_stamp: "Formatted at 2026-09-28 22:54:28 on dist-test-slave-kzfr"
server_key: "721f4088e259afe139ce043e1f1c8126"
server_key_iv: "a4077f59ef833935fd68c5102645979b"
server_key_version: "encryptionkey@0"
I20260928 22:54:28.318137 24162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:28.321403 24162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:28.321734 24162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:28.321877 24162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:28.322078 24162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:28.322417 24162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:28.322463 24162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:28.322502 24162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:28.322530 24162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:28.328625 24162 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:37177
I20260928 22:54:28.328678 24289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:37177 every 8 connection(s)
I20260928 22:54:28.329005 24162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:54:28.332132 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:28.329749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:49303 (local address 127.14.14.190:39929)
0928 22:54:28.329810 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:28.329812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:28.330133 (+ 321us) server_negotiation.cc:407] Connection header received
0928 22:54:28.330624 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:28.330626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:28.330633 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:28.330698 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:28.331895 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:28.331897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:28.331897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:54:28.331902 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:28.331918 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:54:28.331922 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:28.331975 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:28.332047 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 22:54:28.332063 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:54:28.332350 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:28.329775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:39929 (local address 127.14.14.129:49303)
0928 22:54:28.329935 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:28.329969 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:28.330488 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:28.330766 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:28.330770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:28.330779 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:28.331733 (+ 954us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:54:28.331734 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:28.331761 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:54:28.331787 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:28.331972 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:28.331976 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:28.331979 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:54:28.332057 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:54:28.332121 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
I20260928 22:54:28.332675 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 24162
I20260928 22:54:28.332777 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:54:28.332844 24290 heartbeater.cc:347] Connected to a master server at 127.14.14.190:39929
I20260928 22:54:28.332930 24290 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:28.333005 14394 external_mini_cluster.cc:1428] Setting key 58356aa2c87385cb13e42e143536ab0c
I20260928 22:54:28.333055 24290 heartbeater.cc:511] Master 127.14.14.190:39929 requested a full tablet report, sending...
I20260928 22:54:28.333508 24102 ts_manager.cc:194] Registered new tserver with Master: 076b4ed0fa8c48b1b7bfca8eed4819ae (127.14.14.129:37177)
I20260928 22:54:28.334195 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:39929
--builtin_ntp_servers=127.14.14.148:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 22:54:28.415939 24293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:28.416134 24293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:28.416157 24293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:28.417225 24293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:28.417258 24293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:28.417284 24293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:54:28.418632 24293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:39929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:28.418812 24293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:28.419047 24293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:28.424090 24301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:28.424221 24293 server_base.cc:1034] running on GCE node
W20260928 22:54:28.424119 24298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:28.424456 24299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:28.424736 24293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:28.425050 24293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:28.426239 24293 hybrid_clock.cc:648] HybridClock initialized: now 1790636068426225 us; error 49 us; skew 500 ppm
I20260928 22:54:28.428017 24293 webserver.cc:460] Webserver started at http://127.14.14.130:46359/ using document root <none> and password file <none>
I20260928 22:54:28.428349 24293 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:28.428422 24293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:28.428572 24293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:28.430114 24293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "219718787a5541c29f56727b2ce9bed0"
format_stamp: "Formatted at 2026-09-28 22:54:28 on dist-test-slave-kzfr"
server_key: "8cfc5fa164c4d347124d52987ada2ea6"
server_key_iv: "2521c6515d69fba49dd0218bd695b924"
server_key_version: "encryptionkey@0"
I20260928 22:54:28.430528 24293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "219718787a5541c29f56727b2ce9bed0"
format_stamp: "Formatted at 2026-09-28 22:54:28 on dist-test-slave-kzfr"
server_key: "8cfc5fa164c4d347124d52987ada2ea6"
server_key_iv: "2521c6515d69fba49dd0218bd695b924"
server_key_version: "encryptionkey@0"
I20260928 22:54:28.432054 24293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:54:28.433117 24307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:28.433284 24293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:28.433380 24293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "219718787a5541c29f56727b2ce9bed0"
format_stamp: "Formatted at 2026-09-28 22:54:28 on dist-test-slave-kzfr"
server_key: "8cfc5fa164c4d347124d52987ada2ea6"
server_key_iv: "2521c6515d69fba49dd0218bd695b924"
server_key_version: "encryptionkey@0"
I20260928 22:54:28.433461 24293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:28.436643 24293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:28.436892 24293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:28.437021 24293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:28.437238 24293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:28.437520 24293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:28.437614 24293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:28.437655 24293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:28.437683 24293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:28.443362 24293 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:37799
I20260928 22:54:28.443406 24420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:37799 every 8 connection(s)
I20260928 22:54:28.443688 24293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:54:28.446393 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:28.444295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:48623 (local address 127.14.14.190:39929)
0928 22:54:28.444352 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:28.444354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:28.444580 (+ 226us) server_negotiation.cc:407] Connection header received
0928 22:54:28.445007 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:28.445008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:28.445013 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:28.445065 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:28.446217 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:28.446218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:28.446218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:54:28.446222 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:28.446236 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:54:28.446240 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:28.446288 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:28.446331 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 22:54:28.446342 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:54:28.446606 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:28.444295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:39929 (local address 127.14.14.130:48623)
0928 22:54:28.444473 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:28.444504 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:28.444926 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:28.445128 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:28.445132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:28.445142 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:28.446088 (+ 946us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:54:28.446089 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:28.446116 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:54:28.446142 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:28.446285 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:28.446287 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:28.446289 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:54:28.446336 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 22:54:28.446379 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260928 22:54:28.447062 24421 heartbeater.cc:347] Connected to a master server at 127.14.14.190:39929
I20260928 22:54:28.447168 24421 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:28.447311 24421 heartbeater.cc:511] Master 127.14.14.190:39929 requested a full tablet report, sending...
I20260928 22:54:28.447678 24102 ts_manager.cc:194] Registered new tserver with Master: 219718787a5541c29f56727b2ce9bed0 (127.14.14.130:37799)
I20260928 22:54:28.448632 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 24293
I20260928 22:54:28.448709 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:54:28.448864 14394 external_mini_cluster.cc:1428] Setting key a6d6758b4eeef96d386778b250f0048c
I20260928 22:54:28.449836 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:39929
--builtin_ntp_servers=127.14.14.148:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 22:54:28.530954 24424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:28.531164 24424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:28.531186 24424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:28.532336 24424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:28.532375 24424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:28.532400 24424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:54:28.533695 24424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:39929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:28.533871 24424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:28.534098 24424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:28.539156 24429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:28.539156 24432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:28.539158 24430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:28.539234 24424 server_base.cc:1034] running on GCE node
I20260928 22:54:28.539592 24424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:28.539825 24424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:28.540983 24424 hybrid_clock.cc:648] HybridClock initialized: now 1790636068540979 us; error 36 us; skew 500 ppm
I20260928 22:54:28.542217 24424 webserver.cc:460] Webserver started at http://127.14.14.131:34521/ using document root <none> and password file <none>
I20260928 22:54:28.542465 24424 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:28.542532 24424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:28.542695 24424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:28.543745 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "88923ad08a9b4e05b4a5c0c45cc2e8a9"
format_stamp: "Formatted at 2026-09-28 22:54:28 on dist-test-slave-kzfr"
server_key: "4b5dc3077bcf7335f050b9213b56df53"
server_key_iv: "37e36e52e74556ada1ed41552fc9557f"
server_key_version: "encryptionkey@0"
I20260928 22:54:28.544137 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "88923ad08a9b4e05b4a5c0c45cc2e8a9"
format_stamp: "Formatted at 2026-09-28 22:54:28 on dist-test-slave-kzfr"
server_key: "4b5dc3077bcf7335f050b9213b56df53"
server_key_iv: "37e36e52e74556ada1ed41552fc9557f"
server_key_version: "encryptionkey@0"
I20260928 22:54:28.545419 24424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:54:28.546214 24438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:28.546389 24424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:54:28.546482 24424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "88923ad08a9b4e05b4a5c0c45cc2e8a9"
format_stamp: "Formatted at 2026-09-28 22:54:28 on dist-test-slave-kzfr"
server_key: "4b5dc3077bcf7335f050b9213b56df53"
server_key_iv: "37e36e52e74556ada1ed41552fc9557f"
server_key_version: "encryptionkey@0"
I20260928 22:54:28.546564 24424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:28.549872 24424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:28.550132 24424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:28.550282 24424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:28.550482 24424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:28.550770 24424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:28.550820 24424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:28.550859 24424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:28.550887 24424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:28.556478 24424 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:38035
I20260928 22:54:28.556571 24551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:38035 every 8 connection(s)
I20260928 22:54:28.556867 24424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:54:28.559602 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:28.557528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:57187 (local address 127.14.14.190:39929)
0928 22:54:28.557572 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:28.557574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:28.557856 (+ 282us) server_negotiation.cc:407] Connection header received
0928 22:54:28.558287 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:28.558289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:28.558295 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:28.558348 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:28.559398 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:28.559400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:28.559400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:54:28.559404 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:28.559419 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:54:28.559423 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:28.559477 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:28.559534 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 22:54:28.559549 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:54:28.559791 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:28.557510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:39929 (local address 127.14.14.131:57187)
0928 22:54:28.557734 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:28.557769 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:28.558203 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:28.558398 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:28.558402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:28.558410 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:28.559232 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:54:28.559232 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:28.559280 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:54:28.559307 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:28.559476 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:28.559478 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:28.559481 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:54:28.559530 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 22:54:28.559574 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":140,"threads_started":1}
I20260928 22:54:28.560253 24552 heartbeater.cc:347] Connected to a master server at 127.14.14.190:39929
I20260928 22:54:28.560345 24552 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:28.560489 24552 heartbeater.cc:511] Master 127.14.14.190:39929 requested a full tablet report, sending...
I20260928 22:54:28.560848 24102 ts_manager.cc:194] Registered new tserver with Master: 88923ad08a9b4e05b4a5c0c45cc2e8a9 (127.14.14.131:38035)
I20260928 22:54:28.564206 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 24424
I20260928 22:54:28.564283 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:54:28.564517 14394 external_mini_cluster.cc:1428] Setting key 6177e92d51e5591fda7a930b117cf579
I20260928 22:54:28.565449 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 22:54:28.567301 24142 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:54:28.566785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.14.14.190:39929)
0928 22:54:28.566836 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:28.566838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:28.566935 (+ 97us) server_negotiation.cc:407] Connection header received
0928 22:54:28.566984 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:28.566985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:28.566988 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:28.567036 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:28.567225 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56462: BlockingRecv error: recv got EOF from 127.0.0.1:56462 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:54:28.567312 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 24162
I20260928 22:54:28.572718 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 24293
I20260928 22:54:28.576874 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 24424
I20260928 22:54:28.581146 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 24071
2026-09-28T22:54:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (528 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:54:28 dist-test-slave-kzfr krb5kdc[24562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:54:28 dist-test-slave-kzfr krb5kdc[24562](info): set up 2 sockets
Sep 28 22:54:28 dist-test-slave-kzfr krb5kdc[24562](info): commencing operation
krb5kdc: starting...
W20260928 22:54:29.630875 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:54:29 dist-test-slave-kzfr krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:54:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:54:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:54:29.750965 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:37529
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:37529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:29.835086 24575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:29.835330 24575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:29.835366 24575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:29.836496 24575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:54:29.836557 24575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:29.836582 24575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:29.836603 24575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:54:29.836627 24575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:54:29.838109 24575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:37529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:37529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:29.838382 24575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:29.838603 24575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:29.843657 24581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:29.843710 24575 server_base.cc:1034] running on GCE node
W20260928 22:54:29.843657 24583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:29.843676 24580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:29.844188 24575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:29.844476 24575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:29.845656 24575 hybrid_clock.cc:648] HybridClock initialized: now 1790636069845638 us; error 38 us; skew 500 ppm
Sep 28 22:54:29 dist-test-slave-kzfr krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636069, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:29.853761 24575 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:54:29.854199 24575 webserver.cc:460] Webserver started at http://127.14.14.190:40077/ using document root <none> and password file <none>
I20260928 22:54:29.854409 24575 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:29.854451 24575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:29.854540 24575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:29.855700 24575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "152420a0b37641a5b45bfacf501c3159"
format_stamp: "Formatted at 2026-09-28 22:54:29 on dist-test-slave-kzfr"
server_key: "097313042a209577966ab532b2454853"
server_key_iv: "73f073a2d95ddf5edd49a702c2a60b63"
server_key_version: "encryptionkey@0"
I20260928 22:54:29.856009 24575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "152420a0b37641a5b45bfacf501c3159"
format_stamp: "Formatted at 2026-09-28 22:54:29 on dist-test-slave-kzfr"
server_key: "097313042a209577966ab532b2454853"
server_key_iv: "73f073a2d95ddf5edd49a702c2a60b63"
server_key_version: "encryptionkey@0"
I20260928 22:54:29.857384 24575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:54:29.858309 24590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:29.858493 24575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:54:29.858583 24575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "152420a0b37641a5b45bfacf501c3159"
format_stamp: "Formatted at 2026-09-28 22:54:29 on dist-test-slave-kzfr"
server_key: "097313042a209577966ab532b2454853"
server_key_iv: "73f073a2d95ddf5edd49a702c2a60b63"
server_key_version: "encryptionkey@0"
I20260928 22:54:29.858683 24575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:29.869439 24575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:29.871476 24575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:29.871644 24575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:29.875633 24575 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:37529
I20260928 22:54:29.875662 24642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:37529 every 8 connection(s)
I20260928 22:54:29.876008 24575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:54:29.876997 24643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:29.879371 24643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159: Bootstrap starting.
I20260928 22:54:29.879951 24643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:29.880239 24643 log.cc:824] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:29.881156 24643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159: No bootstrap required, opened a new log
I20260928 22:54:29.882822 24643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152420a0b37641a5b45bfacf501c3159" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37529 } }
I20260928 22:54:29.882962 24643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:29.882985 24643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152420a0b37641a5b45bfacf501c3159, State: Initialized, Role: FOLLOWER
I20260928 22:54:29.883152 24643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152420a0b37641a5b45bfacf501c3159" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37529 } }
I20260928 22:54:29.883224 24643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:54:29.883245 24643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:54:29.883320 24643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:29.884115 24643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152420a0b37641a5b45bfacf501c3159" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37529 } }
I20260928 22:54:29.884202 24643 leader_election.cc:302] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 152420a0b37641a5b45bfacf501c3159; no voters:
I20260928 22:54:29.884389 24643 leader_election.cc:288] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:54:29.884457 24646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:54:29.884617 24646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [term 1 LEADER]: Becoming Leader. State: Replica: 152420a0b37641a5b45bfacf501c3159, State: Running, Role: LEADER
I20260928 22:54:29.884662 24643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:54:29.884792 24646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152420a0b37641a5b45bfacf501c3159" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37529 } }
I20260928 22:54:29.885177 24647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "152420a0b37641a5b45bfacf501c3159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152420a0b37641a5b45bfacf501c3159" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37529 } } }
I20260928 22:54:29.885188 24648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 152420a0b37641a5b45bfacf501c3159. Latest consensus state: current_term: 1 leader_uuid: "152420a0b37641a5b45bfacf501c3159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152420a0b37641a5b45bfacf501c3159" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37529 } } }
I20260928 22:54:29.885342 24647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:29.885351 24648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159 [sys.catalog]: This master's current role is: LEADER
I20260928 22:54:29.885623 24655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:54:29.885943 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 24575
I20260928 22:54:29.886068 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:54:29.886255 24655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:54:29.886273 14394 external_mini_cluster.cc:1428] Setting key 2359392e000abf5dbc409f18986f6279
I20260928 22:54:29.888197 24655 catalog_manager.cc:1347] Generated new cluster ID: 65927791b89b45eb845a7c9bbe2e4ef8
I20260928 22:54:29.888237 24655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:54:29 dist-test-slave-kzfr krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:54:29.899458 24655 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:54:29.900125 24655 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:54:29.900779 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:29.886802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45034 (local address 127.14.14.190:37529)
0928 22:54:29.889792 (+ 2990us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:29.889801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:29.889836 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:54:29.889947 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:29.889953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:29.890006 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:29.890143 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:29.890929 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:29.892311 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:29.893475 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:29.894086 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:29.894177 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:29.896971 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:29.896975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:29.896976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:29.896988 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:29.899033 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:29.899479 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:29.899482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:29.899483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:29.899587 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:29.899804 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:29.899805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:29.899805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:29.900128 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:29.900274 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:29.900421 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 22:54:29.900558 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":2931,"thread_start_us":86,"threads_started":1}
I20260928 22:54:29.904305 24655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159: Generated new TSK 0
I20260928 22:54:29.904466 24655 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:54:29.932214 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:37529
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:30.014093 24669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:30.014293 24669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:30.014317 24669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:30.015475 24669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:30.015522 24669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:30.015549 24669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:54:30.017079 24669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:37529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:30.017329 24669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:30.017561 24669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:54:30.023113 24669 server_base.cc:1034] running on GCE node
W20260928 22:54:30.023160 24674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:30.023036 24677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:30.023034 24675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:30.023447 24669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:30.023691 24669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:30.024850 24669 hybrid_clock.cc:648] HybridClock initialized: now 1790636070024839 us; error 42 us; skew 500 ppm
Sep 28 22:54:30 dist-test-slave-kzfr krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636070, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:30.027442 24669 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:54:30.027869 24669 webserver.cc:460] Webserver started at http://127.14.14.129:44171/ using document root <none> and password file <none>
I20260928 22:54:30.028175 24669 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:30.028249 24669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:30.028390 24669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:30.029424 24669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "d789509265b2410b977f62f66dfce62b"
format_stamp: "Formatted at 2026-09-28 22:54:30 on dist-test-slave-kzfr"
server_key: "b818bab3fd519f85626b9088b3aa5cff"
server_key_iv: "d3f2f881b402d1e0b9e8dc999047e907"
server_key_version: "encryptionkey@0"
I20260928 22:54:30.029807 24669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "d789509265b2410b977f62f66dfce62b"
format_stamp: "Formatted at 2026-09-28 22:54:30 on dist-test-slave-kzfr"
server_key: "b818bab3fd519f85626b9088b3aa5cff"
server_key_iv: "d3f2f881b402d1e0b9e8dc999047e907"
server_key_version: "encryptionkey@0"
I20260928 22:54:30.031131 24669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:54:30.031843 24684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:30.032015 24669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:54:30.032114 24669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "d789509265b2410b977f62f66dfce62b"
format_stamp: "Formatted at 2026-09-28 22:54:30 on dist-test-slave-kzfr"
server_key: "b818bab3fd519f85626b9088b3aa5cff"
server_key_iv: "d3f2f881b402d1e0b9e8dc999047e907"
server_key_version: "encryptionkey@0"
I20260928 22:54:30.032258 24669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:30.045087 24669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:30.046955 24669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:30.047108 24669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:30.047330 24669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:30.047636 24669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:30.047686 24669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:30.047729 24669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:30.047768 24669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:30.053568 24669 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:39729
I20260928 22:54:30.053655 24797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:39729 every 8 connection(s)
I20260928 22:54:30.053972 24669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:54:30.056974 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 24669
I20260928 22:54:30.057085 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:54:30.057403 14394 external_mini_cluster.cc:1428] Setting key 92329099d77bb5af4841baa2998076d5
Sep 28 22:54:30 dist-test-slave-kzfr krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636070, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:54:30.064536 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.054656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:45223 (local address 127.14.14.190:37529)
0928 22:54:30.054719 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:30.054721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:30.054904 (+ 183us) server_negotiation.cc:407] Connection header received
0928 22:54:30.055490 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:30.055491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:30.055501 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:30.055554 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:30.056452 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.056967 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.058403 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.058600 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.058658 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.061378 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:30.061379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:30.061380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:30.061387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:30.062987 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:30.063500 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:30.063501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:30.063501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:30.063550 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:30.063890 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:30.063891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:30.063891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:30.063999 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:30.064053 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:30.064390 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 22:54:30.064469 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:54:30.064817 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.054642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:37529 (local address 127.14.14.129:45223)
0928 22:54:30.054792 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:30.054826 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:30.055403 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:30.055608 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:30.055611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:30.056184 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:30.056375 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.056380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.057057 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.057058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.058325 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.058328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.058682 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.058683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.058842 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.059364 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:30.059375 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:30.061253 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:30.063072 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:30.063080 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:30.063083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:30.063422 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:30.063615 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:30.063617 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:30.063617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:30.063778 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:30.064067 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:30.064074 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:30.064247 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 22:54:30.064370 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 22:54:30.064549 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260928 22:54:30.065359 24798 heartbeater.cc:347] Connected to a master server at 127.14.14.190:37529
I20260928 22:54:30.065459 24798 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:30.065677 24798 heartbeater.cc:511] Master 127.14.14.190:37529 requested a full tablet report, sending...
I20260928 22:54:30.066216 24607 ts_manager.cc:194] Registered new tserver with Master: d789509265b2410b977f62f66dfce62b (127.14.14.129:39729)
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
I20260928 22:54:30.066893 24607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:45223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:54:30.084929 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:37529
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:30.167832 24803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:30.168052 24803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:30.168076 24803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:30.169250 24803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:30.169294 24803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:30.169320 24803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:54:30.170804 24803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:37529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:30.171068 24803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:30.171308 24803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:54:30.176456 24809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:30.176651 24803 server_base.cc:1034] running on GCE node
W20260928 22:54:30.176469 24811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:30.176569 24808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:30.176967 24803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:30.177220 24803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:30.178366 24803 hybrid_clock.cc:648] HybridClock initialized: now 1790636070178344 us; error 38 us; skew 500 ppm
Sep 28 22:54:30 dist-test-slave-kzfr krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636070, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:30.181034 24803 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:54:30.181489 24803 webserver.cc:460] Webserver started at http://127.14.14.130:44363/ using document root <none> and password file <none>
I20260928 22:54:30.181772 24803 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:30.181843 24803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:30.181986 24803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:30.183050 24803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "71472e56fcae49bf9eef59ea054c78b7"
format_stamp: "Formatted at 2026-09-28 22:54:30 on dist-test-slave-kzfr"
server_key: "64c484e50633faf2b1654f2e1c4a3f85"
server_key_iv: "2358c516acf01d7529d65a5d1dd50e9e"
server_key_version: "encryptionkey@0"
I20260928 22:54:30.183399 24803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "71472e56fcae49bf9eef59ea054c78b7"
format_stamp: "Formatted at 2026-09-28 22:54:30 on dist-test-slave-kzfr"
server_key: "64c484e50633faf2b1654f2e1c4a3f85"
server_key_iv: "2358c516acf01d7529d65a5d1dd50e9e"
server_key_version: "encryptionkey@0"
I20260928 22:54:30.184737 24803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:54:30.185607 24818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:30.185774 24803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:54:30.185866 24803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "71472e56fcae49bf9eef59ea054c78b7"
format_stamp: "Formatted at 2026-09-28 22:54:30 on dist-test-slave-kzfr"
server_key: "64c484e50633faf2b1654f2e1c4a3f85"
server_key_iv: "2358c516acf01d7529d65a5d1dd50e9e"
server_key_version: "encryptionkey@0"
I20260928 22:54:30.185943 24803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:30.199103 24803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:30.201256 24803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:30.201448 24803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:30.201798 24803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:30.202128 24803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:30.202183 24803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:30.202225 24803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:30.202252 24803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:30.208066 24803 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:37189
I20260928 22:54:30.208127 24931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:37189 every 8 connection(s)
I20260928 22:54:30.208458 24803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:54:30.210067 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 24803
I20260928 22:54:30.210199 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:54:30.210412 14394 external_mini_cluster.cc:1428] Setting key 4eeeaecf2c19d0d89b4f6504366015af
Sep 28 22:54:30 dist-test-slave-kzfr krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636070, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:54:30.218600 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.209239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:35065 (local address 127.14.14.190:37529)
0928 22:54:30.209315 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:30.209318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:30.209459 (+ 141us) server_negotiation.cc:407] Connection header received
0928 22:54:30.209958 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:30.209960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:30.209972 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:30.210049 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:30.211219 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.212003 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.213014 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.213354 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.213433 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.215555 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:30.215557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:30.215557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:30.215566 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:30.217227 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:30.217621 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:30.217623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:30.217623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:30.217688 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:30.217923 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:30.217923 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:30.217924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:30.218087 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:30.218165 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:30.218410 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 22:54:30.218498 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:54:30.218631 24934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.209191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:37529 (local address 127.14.14.130:35065)
0928 22:54:30.209343 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:30.209378 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:30.209867 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:30.210116 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:30.210119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:30.210873 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:30.211123 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.211129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.212097 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.212098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.212930 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.212932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.213442 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.213443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.213539 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.214130 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:30.214138 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:30.215464 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:30.217309 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:30.217314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:30.217316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:30.217521 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:30.217764 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:30.217765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:30.217765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:30.217860 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:30.218156 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:30.218161 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:30.218280 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 22:54:30.218363 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:54:30.218475 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260928 22:54:30.219115 24932 heartbeater.cc:347] Connected to a master server at 127.14.14.190:37529
I20260928 22:54:30.219213 24932 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
I20260928 22:54:30.219412 24932 heartbeater.cc:511] Master 127.14.14.190:37529 requested a full tablet report, sending...
I20260928 22:54:30.219789 24607 ts_manager.cc:194] Registered new tserver with Master: 71472e56fcae49bf9eef59ea054c78b7 (127.14.14.130:37189)
I20260928 22:54:30.220165 24607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:35065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:54:30.237135 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:37529
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:54:30.318173 24937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:54:30.318379 24937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:54:30.318401 24937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:54:30.319476 24937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:54:30.319509 24937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:54:30.319533 24937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:54:30.320889 24937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:37529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:54:30.321084 24937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:54:30.321314 24937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:54:30.326678 24937 server_base.cc:1034] running on GCE node
W20260928 22:54:30.326609 24945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:30.326586 24943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:54:30.326591 24942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:54:30.326987 24937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:54:30.327278 24937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:54:30.328430 24937 hybrid_clock.cc:648] HybridClock initialized: now 1790636070328409 us; error 38 us; skew 500 ppm
Sep 28 22:54:30 dist-test-slave-kzfr krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636070, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:54:30.331043 24937 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:54:30.331526 24937 webserver.cc:460] Webserver started at http://127.14.14.131:39779/ using document root <none> and password file <none>
I20260928 22:54:30.331729 24937 fs_manager.cc:359] Metadata directory not provided
I20260928 22:54:30.331771 24937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:54:30.331859 24937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:54:30.332875 24937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "35692e82e137402abc94c4f4d1033abd"
format_stamp: "Formatted at 2026-09-28 22:54:30 on dist-test-slave-kzfr"
server_key: "afd6fefd468e3b92e358fd8efb35cd76"
server_key_iv: "190e5e0b80eb08a579df65f35eaecc2f"
server_key_version: "encryptionkey@0"
I20260928 22:54:30.333189 24937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "35692e82e137402abc94c4f4d1033abd"
format_stamp: "Formatted at 2026-09-28 22:54:30 on dist-test-slave-kzfr"
server_key: "afd6fefd468e3b92e358fd8efb35cd76"
server_key_iv: "190e5e0b80eb08a579df65f35eaecc2f"
server_key_version: "encryptionkey@0"
I20260928 22:54:30.334579 24937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:54:30.335384 24952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:30.335546 24937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:54:30.335633 24937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "35692e82e137402abc94c4f4d1033abd"
format_stamp: "Formatted at 2026-09-28 22:54:30 on dist-test-slave-kzfr"
server_key: "afd6fefd468e3b92e358fd8efb35cd76"
server_key_iv: "190e5e0b80eb08a579df65f35eaecc2f"
server_key_version: "encryptionkey@0"
I20260928 22:54:30.335726 24937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:54:30.359597 24937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:54:30.361984 24937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:54:30.362131 24937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:54:30.362393 24937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:54:30.362699 24937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:54:30.362731 24937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:30.362771 24937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:54:30.362798 24937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:54:30.368800 24937 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:36671
I20260928 22:54:30.368849 25065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:36671 every 8 connection(s)
I20260928 22:54:30.369158 24937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:54:30.372116 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 24937
I20260928 22:54:30.372233 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:54:30.372411 14394 external_mini_cluster.cc:1428] Setting key 85fcd4d76ca411b8c972d7a4d11fe75c
Sep 28 22:54:30 dist-test-slave-kzfr krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636070, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:54:30.379405 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.369871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:48599 (local address 127.14.14.190:37529)
0928 22:54:30.369937 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:30.369940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:30.370210 (+ 270us) server_negotiation.cc:407] Connection header received
0928 22:54:30.371253 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:30.371254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:30.371265 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:30.371321 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:30.371975 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.372836 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.374068 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.374254 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.374305 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.376515 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:30.376517 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:30.376517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:30.376524 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:30.378035 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:30.378480 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:30.378481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:30.378481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:30.378540 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:30.378794 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:30.378794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:30.378795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:30.378905 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:30.378963 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:30.379272 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 22:54:30.379338 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:54:30.379657 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.369867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:37529 (local address 127.14.14.131:48599)
0928 22:54:30.370041 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:30.370094 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:30.371067 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:30.371394 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:30.371397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:30.371681 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:54:30.371874 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.371879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.372972 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.372973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.373989 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.373991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.374326 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.374327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.374432 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.374906 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:54:30.374915 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:54:30.376426 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:54:30.378151 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:30.378158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:30.378160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:30.378393 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:30.378612 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:54:30.378613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:54:30.378613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:54:30.378722 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:54:30.378971 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:54:30.378976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:54:30.379096 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 22:54:30.379238 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 22:54:30.379378 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260928 22:54:30.380203 25066 heartbeater.cc:347] Connected to a master server at 127.14.14.190:37529
I20260928 22:54:30.380326 25066 heartbeater.cc:464] Registering TS with master...
I20260928 22:54:30.380475 25066 heartbeater.cc:511] Master 127.14.14.190:37529 requested a full tablet report, sending...
I20260928 22:54:30.380851 24607 ts_manager.cc:194] Registered new tserver with Master: 35692e82e137402abc94c4f4d1033abd (127.14.14.131:36671)
I20260928 22:54:30.381278 24607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:48599
I20260928 22:54:30.384004 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:54:30.391637 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.385470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.14.14.190:37529)
0928 22:54:30.385542 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:30.385544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:30.385719 (+ 175us) server_negotiation.cc:407] Connection header received
0928 22:54:30.385760 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:30.385761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:30.385770 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:30.385848 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:54:30.386467 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.387091 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.387811 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.387996 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.388056 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.389285 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:54:30.389286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:54:30.389286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:54:30.389292 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:54:30.390637 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:30.390945 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:30.390946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:30.390946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:30.390996 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:54:30.391215 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:54:30.391215 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:54:30.391215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:54:30.391349 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:54:30.391422 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:30.391523 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 22:54:30.391577 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:54:30.393116 24607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:54:30.394778 24607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:54:30.401613 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.396873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:37189 (local address 127.0.0.1:50444)
0928 22:54:30.397194 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:30.397204 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:30.397298 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:30.397643 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:30.397646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:30.397648 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:30.397709 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.397711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.399125 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.399126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.400212 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.400215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.401286 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.401286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.401371 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.401387 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 22:54:30.401460 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:54:30.401512 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":2,"thread_start_us":53,"threads_started":1}
I20260928 22:54:30.401573 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.396964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.14.14.130:37189)
0928 22:54:30.397132 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:30.397138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:30.397269 (+ 131us) server_negotiation.cc:407] Connection header received
0928 22:54:30.397405 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:30.397408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:30.397427 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:30.397478 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:30.397782 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.399012 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.400319 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.401219 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.401283 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.401333 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:30.401476 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 22:54:30.401510 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260928 22:54:30.402336 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.396867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:39729 (local address 127.0.0.1:39202)
0928 22:54:30.397011 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:30.397060 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:30.397120 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:30.398016 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:30.398017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:30.398019 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:30.398073 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.398075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.398901 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.398902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.400212 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.400215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.402096 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.402096 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.402167 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.402170 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:54:30.402220 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 22:54:30.402269 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":7,"thread_start_us":73,"threads_started":1}
I20260928 22:54:30.402364 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.397169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.14.14.129:39729)
0928 22:54:30.397864 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:30.397868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:30.397877 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:54:30.397912 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:30.397914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:30.397933 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:30.398018 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:30.398132 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.398805 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.400320 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.401093 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.401200 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.401270 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:30.402259 (+ 989us) server_negotiation.cc:299] Negotiation successful
0928 22:54:30.402304 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":73,"threads_started":1}
I20260928 22:54:30.402462 24866 tablet_service.cc:1467] Processing CreateTablet for tablet 70b2af4e36d74875a5db421d20221973 (DEFAULT_TABLE table=test-table [id=7369d110ed314486aac103c33b097106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:30.402653 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.397115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:36671 (local address 127.0.0.1:46800)
0928 22:54:30.397336 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:30.397346 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:30.397389 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:30.397752 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:30.397753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:30.397756 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:30.397807 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.397809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.400318 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.400319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.401100 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.401102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.402427 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.402427 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.402494 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.402497 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:54:30.402557 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:54:30.402592 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":48,"threads_started":1}
I20260928 22:54:30.402794 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.397276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.14.14.131:36671)
0928 22:54:30.397439 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:30.397443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:30.397453 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:54:30.397534 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:30.397536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:30.397553 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:30.397674 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:30.397892 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.399012 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.401187 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.401943 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.402009 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.402062 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:30.402579 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 22:54:30.402621 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
I20260928 22:54:30.402832 24866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b2af4e36d74875a5db421d20221973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:30.403329 25000 tablet_service.cc:1467] Processing CreateTablet for tablet 70b2af4e36d74875a5db421d20221973 (DEFAULT_TABLE table=test-table [id=7369d110ed314486aac103c33b097106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:30.403614 25000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b2af4e36d74875a5db421d20221973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:30.405650 25080 tablet_bootstrap.cc:492] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7: Bootstrap starting.
I20260928 22:54:30.406113 24732 tablet_service.cc:1467] Processing CreateTablet for tablet 70b2af4e36d74875a5db421d20221973 (DEFAULT_TABLE table=test-table [id=7369d110ed314486aac103c33b097106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:54:30.406365 25080 tablet_bootstrap.cc:654] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:30.406424 24732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b2af4e36d74875a5db421d20221973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:54:30.406446 25081 tablet_bootstrap.cc:492] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd: Bootstrap starting.
I20260928 22:54:30.406970 25080 log.cc:824] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:30.407160 25081 tablet_bootstrap.cc:654] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:30.407671 25081 log.cc:824] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:30.407809 25080 tablet_bootstrap.cc:492] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7: No bootstrap required, opened a new log
I20260928 22:54:30.407884 25080 ts_tablet_manager.cc:1397] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:54:30.409211 25084 tablet_bootstrap.cc:492] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b: Bootstrap starting.
I20260928 22:54:30.409363 25080 raft_consensus.cc:348] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71472e56fcae49bf9eef59ea054c78b7" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37189 } } peers { permanent_uuid: "d789509265b2410b977f62f66dfce62b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39729 } } peers { permanent_uuid: "35692e82e137402abc94c4f4d1033abd" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36671 } }
I20260928 22:54:30.409514 25080 raft_consensus.cc:374] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:30.409538 25080 raft_consensus.cc:729] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71472e56fcae49bf9eef59ea054c78b7, State: Initialized, Role: FOLLOWER
I20260928 22:54:30.409752 25080 consensus_queue.cc:260] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71472e56fcae49bf9eef59ea054c78b7" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37189 } } peers { permanent_uuid: "d789509265b2410b977f62f66dfce62b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39729 } } peers { permanent_uuid: "35692e82e137402abc94c4f4d1033abd" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36671 } }
I20260928 22:54:30.409861 25084 tablet_bootstrap.cc:654] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b: Neither blocks nor log segments found. Creating new log.
I20260928 22:54:30.410034 25080 ts_tablet_manager.cc:1428] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:54:30.410085 24932 heartbeater.cc:503] Master 127.14.14.190:37529 was elected leader, sending a full tablet report...
I20260928 22:54:30.410396 25084 log.cc:824] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:54:30.410552 25081 tablet_bootstrap.cc:492] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd: No bootstrap required, opened a new log
I20260928 22:54:30.410645 25081 ts_tablet_manager.cc:1397] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 22:54:30.411196 25084 tablet_bootstrap.cc:492] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b: No bootstrap required, opened a new log
I20260928 22:54:30.411271 25084 ts_tablet_manager.cc:1397] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:54:30.412226 25081 raft_consensus.cc:348] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71472e56fcae49bf9eef59ea054c78b7" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37189 } } peers { permanent_uuid: "d789509265b2410b977f62f66dfce62b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39729 } } peers { permanent_uuid: "35692e82e137402abc94c4f4d1033abd" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36671 } }
I20260928 22:54:30.412355 25081 raft_consensus.cc:374] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:30.412376 25081 raft_consensus.cc:729] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35692e82e137402abc94c4f4d1033abd, State: Initialized, Role: FOLLOWER
I20260928 22:54:30.412559 25081 consensus_queue.cc:260] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71472e56fcae49bf9eef59ea054c78b7" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37189 } } peers { permanent_uuid: "d789509265b2410b977f62f66dfce62b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39729 } } peers { permanent_uuid: "35692e82e137402abc94c4f4d1033abd" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36671 } }
I20260928 22:54:30.412708 25084 raft_consensus.cc:348] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71472e56fcae49bf9eef59ea054c78b7" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37189 } } peers { permanent_uuid: "d789509265b2410b977f62f66dfce62b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39729 } } peers { permanent_uuid: "35692e82e137402abc94c4f4d1033abd" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36671 } }
I20260928 22:54:30.412856 25084 raft_consensus.cc:374] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:54:30.412848 25066 heartbeater.cc:503] Master 127.14.14.190:37529 was elected leader, sending a full tablet report...
I20260928 22:54:30.412854 25081 ts_tablet_manager.cc:1428] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:54:30.412892 25084 raft_consensus.cc:729] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d789509265b2410b977f62f66dfce62b, State: Initialized, Role: FOLLOWER
I20260928 22:54:30.412992 25084 consensus_queue.cc:260] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71472e56fcae49bf9eef59ea054c78b7" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37189 } } peers { permanent_uuid: "d789509265b2410b977f62f66dfce62b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39729 } } peers { permanent_uuid: "35692e82e137402abc94c4f4d1033abd" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36671 } }
I20260928 22:54:30.413224 25084 ts_tablet_manager.cc:1428] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:54:30.413342 24798 heartbeater.cc:503] Master 127.14.14.190:37529 was elected leader, sending a full tablet report...
W20260928 22:54:30.459036 24933 tablet.cc:2369] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:54:30.554669 24799 tablet.cc:2369] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:54:30.619796 25067 tablet.cc:2369] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:54:30.819275 25088 raft_consensus.cc:482] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:54:30.819415 25088 raft_consensus.cc:504] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71472e56fcae49bf9eef59ea054c78b7" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37189 } } peers { permanent_uuid: "d789509265b2410b977f62f66dfce62b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39729 } } peers { permanent_uuid: "35692e82e137402abc94c4f4d1033abd" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36671 } }
I20260928 22:54:30.819854 25088 leader_election.cc:288] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71472e56fcae49bf9eef59ea054c78b7 (127.14.14.130:37189), 35692e82e137402abc94c4f4d1033abd (127.14.14.131:36671)
I20260928 22:54:30.823980 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.820403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:57693 (local address 127.14.14.131:36671)
0928 22:54:30.820475 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:30.820478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:30.820488 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:54:30.820551 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:30.820552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:30.820563 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:30.820616 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:30.820768 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.821458 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.823030 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.823604 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.823687 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.823695 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:30.823869 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 22:54:30.823910 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 22:54:30.823980 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.820210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:44943 (local address 127.14.14.130:37189)
0928 22:54:30.820308 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:30.820310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:30.820321 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:54:30.820380 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:30.820381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:30.820392 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:30.820422 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:54:30.820637 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.821428 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.823030 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.823604 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.823687 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.823695 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:30.823869 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 22:54:30.823909 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 22:54:30.824031 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.820056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:37189 (local address 127.14.14.129:44943)
0928 22:54:30.820203 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:30.820213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:30.820305 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:30.820499 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:30.820499 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:30.820501 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:30.820561 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.820562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.821533 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.821534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.822934 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.822937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.823699 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.823700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.823793 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.823795 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:54:30.823850 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:54:30.823906 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":81,"thread_start_us":97,"threads_started":1}
I20260928 22:54:30.824034 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.820294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:36671 (local address 127.14.14.129:57693)
0928 22:54:30.820399 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:30.820414 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:54:30.820492 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:54:30.820639 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:54:30.820640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:54:30.820642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:54:30.820700 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.820701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.821529 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.821530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.822934 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:54:30.822937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.823697 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.823699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:54:30.823784 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.823789 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:54:30.823850 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 22:54:30.823906 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":78,"thread_start_us":56,"threads_started":1}
I20260928 22:54:30.824435 24886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b2af4e36d74875a5db421d20221973" candidate_uuid: "d789509265b2410b977f62f66dfce62b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71472e56fcae49bf9eef59ea054c78b7" is_pre_election: true
I20260928 22:54:30.824455 25020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b2af4e36d74875a5db421d20221973" candidate_uuid: "d789509265b2410b977f62f66dfce62b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35692e82e137402abc94c4f4d1033abd" is_pre_election: true
I20260928 22:54:30.824591 24886 raft_consensus.cc:2454] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d789509265b2410b977f62f66dfce62b in term 0.
I20260928 22:54:30.824618 25020 raft_consensus.cc:2454] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d789509265b2410b977f62f66dfce62b in term 0.
I20260928 22:54:30.824826 24687 leader_election.cc:302] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71472e56fcae49bf9eef59ea054c78b7, d789509265b2410b977f62f66dfce62b; no voters:
I20260928 22:54:30.825029 25088 raft_consensus.cc:2781] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:54:30.825069 25088 raft_consensus.cc:482] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:54:30.825085 25088 raft_consensus.cc:3037] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:30.825878 25088 raft_consensus.cc:504] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71472e56fcae49bf9eef59ea054c78b7" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37189 } } peers { permanent_uuid: "d789509265b2410b977f62f66dfce62b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39729 } } peers { permanent_uuid: "35692e82e137402abc94c4f4d1033abd" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36671 } }
I20260928 22:54:30.826012 25088 leader_election.cc:288] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [CANDIDATE]: Term 1 election: Requested vote from peers 71472e56fcae49bf9eef59ea054c78b7 (127.14.14.130:37189), 35692e82e137402abc94c4f4d1033abd (127.14.14.131:36671)
I20260928 22:54:30.826160 24886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b2af4e36d74875a5db421d20221973" candidate_uuid: "d789509265b2410b977f62f66dfce62b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71472e56fcae49bf9eef59ea054c78b7"
I20260928 22:54:30.826179 25020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b2af4e36d74875a5db421d20221973" candidate_uuid: "d789509265b2410b977f62f66dfce62b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35692e82e137402abc94c4f4d1033abd"
I20260928 22:54:30.826251 25020 raft_consensus.cc:3037] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:30.826258 24886 raft_consensus.cc:3037] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:54:30.827162 25020 raft_consensus.cc:2454] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d789509265b2410b977f62f66dfce62b in term 1.
I20260928 22:54:30.827162 24886 raft_consensus.cc:2454] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d789509265b2410b977f62f66dfce62b in term 1.
I20260928 22:54:30.827332 24687 leader_election.cc:302] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35692e82e137402abc94c4f4d1033abd, d789509265b2410b977f62f66dfce62b; no voters:
I20260928 22:54:30.827425 25088 raft_consensus.cc:2781] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:54:30.827559 25088 raft_consensus.cc:686] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [term 1 LEADER]: Becoming Leader. State: Replica: d789509265b2410b977f62f66dfce62b, State: Running, Role: LEADER
I20260928 22:54:30.827652 25088 consensus_queue.cc:237] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71472e56fcae49bf9eef59ea054c78b7" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37189 } } peers { permanent_uuid: "d789509265b2410b977f62f66dfce62b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39729 } } peers { permanent_uuid: "35692e82e137402abc94c4f4d1033abd" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36671 } }
I20260928 22:54:30.828454 24607 catalog_manager.cc:5495] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b reported cstate change: term changed from 0 to 1, leader changed from <none> to d789509265b2410b977f62f66dfce62b (127.14.14.129). New cstate: current_term: 1 leader_uuid: "d789509265b2410b977f62f66dfce62b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71472e56fcae49bf9eef59ea054c78b7" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d789509265b2410b977f62f66dfce62b" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35692e82e137402abc94c4f4d1033abd" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36671 } health_report { overall_health: UNKNOWN } } }
I20260928 22:54:30.841019 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.838477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39206 (local address 127.14.14.129:39729)
0928 22:54:30.838536 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:54:30.838538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:54:30.838603 (+ 65us) server_negotiation.cc:407] Connection header received
0928 22:54:30.838689 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:54:30.838689 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:54:30.838705 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:54:30.838766 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:54:30.838895 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.839453 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.840328 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:54:30.840502 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:54:30.840549 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:54:30.840760 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:54:30.840856 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:54:30.840917 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:54:30.840950 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 22:54:30.840965 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:54:30.842717 24886 raft_consensus.cc:1261] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7 [term 1 FOLLOWER]: Refusing update from remote peer d789509265b2410b977f62f66dfce62b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:30.842717 25020 raft_consensus.cc:1261] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd [term 1 FOLLOWER]: Refusing update from remote peer d789509265b2410b977f62f66dfce62b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:54:30.843050 25088 consensus_queue.cc:1035] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71472e56fcae49bf9eef59ea054c78b7" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:30.843182 25091 consensus_queue.cc:1035] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35692e82e137402abc94c4f4d1033abd" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 36671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:54:30.845463 25094 mvcc.cc:205] Tried to move back new op lower bound from 7334445346169131008 to 7334445346110672896. Current Snapshot: MvccSnapshot[applied={T|T < 7334445346169131008}]
I20260928 22:54:30.845798 25095 mvcc.cc:205] Tried to move back new op lower bound from 7334445346169131008 to 7334445346110672896. Current Snapshot: MvccSnapshot[applied={T|T < 7334445346169131008}]
I20260928 22:54:30.847949 24607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:54:30.848030 24607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:54:30.848696 24607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 152420a0b37641a5b45bfacf501c3159: Sending DeleteTablet for 3 replicas of tablet 70b2af4e36d74875a5db421d20221973
I20260928 22:54:30.848999 24866 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b2af4e36d74875a5db421d20221973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:30 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:50444
I20260928 22:54:30.849073 25000 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b2af4e36d74875a5db421d20221973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:30 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:46800
I20260928 22:54:30.849221 25104 tablet_replica.cc:330] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7: stopping tablet replica
I20260928 22:54:30.849220 24732 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b2af4e36d74875a5db421d20221973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:54:30 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:39202
I20260928 22:54:30.849313 25104 raft_consensus.cc:2229] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:54:30.849426 25104 raft_consensus.cc:2258] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:30.849548 25106 tablet_replica.cc:330] T 70b2af4e36d74875a5db421d20221973 P d789509265b2410b977f62f66dfce62b: stopping tablet replica
I20260928 22:54:30.849680 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 24669
I20260928 22:54:30.849433 25105 tablet_replica.cc:330] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd: stopping tablet replica
I20260928 22:54:30.850390 25105 raft_consensus.cc:2229] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:54:30.850515 25105 raft_consensus.cc:2258] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:54:30.850829 25104 ts_tablet_manager.cc:1905] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:30.852268 25105 ts_tablet_manager.cc:1905] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:54:30.852483 25104 ts_tablet_manager.cc:1918] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:54:30.852553 25104 log.cc:1196] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/70b2af4e36d74875a5db421d20221973
I20260928 22:54:30.852819 25104 ts_tablet_manager.cc:1939] T 70b2af4e36d74875a5db421d20221973 P 71472e56fcae49bf9eef59ea054c78b7: Deleting consensus metadata
I20260928 22:54:30.853221 24593 catalog_manager.cc:4841] TS 71472e56fcae49bf9eef59ea054c78b7 (127.14.14.130:37189): tablet 70b2af4e36d74875a5db421d20221973 (table test-table [id=7369d110ed314486aac103c33b097106]) successfully deleted
I20260928 22:54:30.853873 25105 ts_tablet_manager.cc:1918] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:54:30.853927 25105 log.cc:1196] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/70b2af4e36d74875a5db421d20221973
I20260928 22:54:30.854158 25105 ts_tablet_manager.cc:1939] T 70b2af4e36d74875a5db421d20221973 P 35692e82e137402abc94c4f4d1033abd: Deleting consensus metadata
I20260928 22:54:30.854642 24593 catalog_manager.cc:4841] TS 35692e82e137402abc94c4f4d1033abd (127.14.14.131:36671): tablet 70b2af4e36d74875a5db421d20221973 (table test-table [id=7369d110ed314486aac103c33b097106]) successfully deleted
W20260928 22:54:30.856058 24594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:39729 (error 108)
I20260928 22:54:30.856509 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:54:30.856396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:39729 (local address 127.0.0.1:39210)
0928 22:54:30.856448 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 22:54:30.856481 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:39729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 22:54:30.856634 24594 catalog_manager.cc:4584] TS d789509265b2410b977f62f66dfce62b (127.14.14.129:39729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70b2af4e36d74875a5db421d20221973: Network error: Client connection negotiation failed: client connection to 127.14.14.129:39729: connect: Connection refused (error 111)
I20260928 22:54:30.856648 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 24803
I20260928 22:54:30.861137 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 24937
I20260928 22:54:30.866755 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 24575
2026-09-28T22:54:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2286 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:54:30 dist-test-slave-kzfr krb5kdc[25110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:54:30 dist-test-slave-kzfr krb5kdc[25110](info): set up 2 sockets
Sep 28 22:54:30 dist-test-slave-kzfr krb5kdc[25110](info): commencing operation
krb5kdc: starting...
W20260928 22:54:31.910718 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:54:31 dist-test-slave-kzfr krb5kdc[25110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:54:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:54:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.14.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.14.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/rangeradmin_127.14.14.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/rangeradmin_127.14.14.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.14.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.14.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/rangerlookup_127.14.14.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/rangerlookup_127.14.14.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.14.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.14.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/HTTP_127.14.14.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/HTTP_127.14.14.148@KRBTEST.COM.keytab.
I20260928 22:54:32.086048 14394 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkdZKSl/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/postgres -l logfile start
2026-09-28 22:54:34.805 UTC [25140] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 22:54:34.805 UTC [25140] LOG: listening on IPv4 address "127.14.14.148", port 37927
2026-09-28 22:54:34.821 UTC [25140] LOG: listening on Unix socket "/tmp/.s.PGSQL.37927"
2026-09-28 22:54:34.843 UTC [25143] LOG: database system was shut down at 2026-09-28 22:54:33 UTC
2026-09-28 22:54:34.846 UTC [25140] LOG: database system is ready to accept connections
I20260928 22:54:35.794823 14394 mini_postgres.cc:96] Postgres bound to 37927
2026-09-28 22:54:35.800 UTC [25151] FATAL: database "slave" does not exist
127.14.14.148:37927 - accepting connections
I20260928 22:54:35.801374 14394 mini_ranger.cc:161] Starting Ranger...
I20260928 22:54:35.820201 14394 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 22:54:36.363046 14394 mini_ranger.cc:87] Created ranger Postgres database
I20260928 22:54:36.363143 14394 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-admin
2026-09-28 22:54:36,648 [I] DB FLAVOR :POSTGRES
2026-09-28 22:54:36,649 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 22:54:36,649 [I] Checking connection..
2026-09-28 22:54:36,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 22:54:36,907 [I] Checking connection passed.
2026-09-28 22:54:36,908 [I] --------- Verifying version history table ---------
2026-09-28 22:54:36,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:54:37,171 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 22:54:37,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:54:37,426 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 22:54:37,426 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 22:54:37,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 22:54:37.685 UTC [25215] WARNING: there is no transaction in progress
2026-09-28 22:54:37,692 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 22:54:37,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:54:37,957 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 22:54:37,958 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 22:54:37,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 22:54:38,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 22:54:38,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kzfr.c.gcp-upstream.internal','N') ;"
2026-09-28 22:54:38,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:54:39,010 [I] Table x_portal_user does not exist in database ranger
2026-09-28 22:54:39,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:54:39,284 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 22:54:39,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:54:39,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:54:39,805 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 22:54:39,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 22:54:40.823 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:40.842 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:40.860 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:40.877 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:40.894 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:40.913 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:40.970 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:40.977 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:40.994 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:41.006 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:41.017 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:41.032 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:41.042 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:41.050 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:41.058 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:41.066 UTC [25323] WARNING: there is no transaction in progress
2026-09-28 22:54:41,772 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 22:54:41,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 22:54:42,030 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 22:54:42,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:54:42,280 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 22:54:42,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:54:42,521 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 22:54:42,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:54:42,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:54:43,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kzfr.c.gcp-upstream.internal';"
2026-09-28 22:54:43,265 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 22:54:43,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 22:54:43,519 [I] DB_PATCHES have already been applied
I20260928 22:54:43.524752 14394 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkdZKSl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 22:54:43.524820 14394 mini_ranger.cc:193] Using host: 127.14.14.148
I20260928 22:54:43.527944 14394 mini_ranger.cc:227] Ranger admin URL: http://127.14.14.148:51305
Sep 28, 2026 10:54:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:54:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:54:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 10:54:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 10:54:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 10:54:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 10:54:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 22:54:44.654284 14394 mini_ranger.cc:160] Time spent starting Ranger: real 8.853s user 0.000s sys 0.009s
Sep 28, 2026 10:55:04 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 10:55:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 10:55:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 10:55:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 10:55:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 10:55:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 10:55:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 10:55:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 10:55:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 10:55:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 10:55:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 10:55:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 10:55:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 10:55:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 10:55:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 10:55:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 10:55:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 10:55:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 22:55:06.280817 14394 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.14.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.14.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/rangerkms_127.14.14.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/rangerkms_127.14.14.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.14.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.14.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.14.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/HTTP_127.14.14.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.14.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/HTTP_127.14.14.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 22:55:06 dist-test-slave-kzfr krb5kdc[25110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636106, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 22:55:06.368602 14394 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 22:55:06.378634 14394 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 22:55:07.397714 14394 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 22:55:07.397802 14394 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-kms
I20260928 22:55:07.397825 14394 mini_ranger_kms.cc:227] Using postgres at 127.14.14.148:37927
2026-09-28 22:55:07,572 [I] DB FLAVOR :POSTGRES
2026-09-28 22:55:07,572 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 22:55:07,572 [I] Checking connection
2026-09-28 22:55:07,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 22:55:07,844 [I] connection success
2026-09-28 22:55:07,844 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 22:55:07,844 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 22:55:07,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 22:55:08,078 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 22:55:08,078 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 22:55:08,078 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 22:55:08,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:37927/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 22:55:08,377 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 22:55:08.856374 14394 mini_ranger_kms.cc:310] Created kms service
I20260928 22:55:09.062987 14394 mini_ranger_kms.cc:326] Created kudu user
I20260928 22:55:09.113425 14394 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 22:55:09 dist-test-slave-kzfr krb5kdc[25110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636106, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
I20260928 22:55:09.515899 14394 mini_ranger_kms.cc:384] Added ranger policy
I20260928 22:55:09.516017 14394 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 22:55:09.516039 14394 mini_ranger_kms.cc:240] Using host: 127.14.14.148
I20260928 22:55:09.519969 14394 mini_ranger_kms.cc:276] Ranger KMS PID: 25503
I20260928 22:55:09.520103 14394 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.14.148:52671
Sep 28, 2026 10:55:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:55:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:55:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 10:55:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 10:55:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 10:55:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 22:55:11.884725 14394 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.516s user 0.013s sys 0.145s
I20260928 22:55:11.884845 14394 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 22:55:11.884938 14394 mini_ranger_kms.cc:401] 127.14.14.148:52671/kms/v1/keys
Sep 28 22:55:12 dist-test-slave-kzfr krb5kdc[25110](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790636112, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.14.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 22:55:12 dist-test-slave-kzfr krb5kdc[25110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636112, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.14.148@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
Sep 28 22:55:13 dist-test-slave-kzfr krb5kdc[25110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636112, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.14.148@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
Sep 28 22:55:15 dist-test-slave-kzfr krb5kdc[25110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:55:15.333199 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:46417
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:46417
--ranger_config_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.14.148:52671/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:55:15.416414 25554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:55:15.416622 25554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:55:15.416646 25554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:55:15.417958 25554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:55:15.417999 25554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:55:15.418013 25554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:55:15.418026 25554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:55:15.418040 25554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:55:15.419598 25554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.14.148:52671/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:46417
--ranger_config_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:46417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:55:15.419837 25554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:55:15.420089 25554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:55:15.426380 25562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:55:15.426395 25560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:55:15.426434 25554 server_base.cc:1034] running on GCE node
W20260928 22:55:15.426553 25559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:55:15.426939 25554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:55:15.427294 25554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:55:15.428476 25554 hybrid_clock.cc:648] HybridClock initialized: now 1790636115428459 us; error 38 us; skew 500 ppm
Sep 28 22:55:15 dist-test-slave-kzfr krb5kdc[25110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636115, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:55:15.436486 25554 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:55:15.437040 25554 webserver.cc:460] Webserver started at http://127.14.14.190:38043/ using document root <none> and password file <none>
I20260928 22:55:15.437268 25554 fs_manager.cc:359] Metadata directory not provided
I20260928 22:55:15.437327 25554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:55:15.437475 25554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:55:15 dist-test-slave-kzfr krb5kdc[25110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636115, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
I20260928 22:55:15.498083 25554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "c4724083886345c99d3869d93bf07f4d"
format_stamp: "Formatted at 2026-09-28 22:55:15 on dist-test-slave-kzfr"
server_key: "15bd40512993d2901340214b9d612878"
server_key_iv: "9479e67e1cb0409100306c8025731085"
server_key_version: "kuduclusterkey@0"
I20260928 22:55:15.498548 25554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "c4724083886345c99d3869d93bf07f4d"
format_stamp: "Formatted at 2026-09-28 22:55:15 on dist-test-slave-kzfr"
server_key: "15bd40512993d2901340214b9d612878"
server_key_iv: "9479e67e1cb0409100306c8025731085"
server_key_version: "kuduclusterkey@0"
I20260928 22:55:15.500437 25554 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 22:55:15.515398 25569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:55:15.515658 25554 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:55:15.515767 25554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "c4724083886345c99d3869d93bf07f4d"
format_stamp: "Formatted at 2026-09-28 22:55:15 on dist-test-slave-kzfr"
server_key: "15bd40512993d2901340214b9d612878"
server_key_iv: "9479e67e1cb0409100306c8025731085"
server_key_version: "kuduclusterkey@0"
I20260928 22:55:15.515892 25554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:55:15.528712 25554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:55:15.530889 25554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:55:15.531057 25554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:55:15.535440 25554 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:46417
I20260928 22:55:15.535512 25621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:46417 every 8 connection(s)
I20260928 22:55:15.535856 25554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:55:15.536599 25622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:55:15.539356 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 25554
I20260928 22:55:15.539491 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:55:15.539615 25622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d: Bootstrap starting.
I20260928 22:55:15.539721 14394 external_mini_cluster.cc:1428] Setting key 3f976a7b03b9f8ba396a0b61b74b0252
I20260928 22:55:15.540277 25622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d: Neither blocks nor log segments found. Creating new log.
I20260928 22:55:15.540577 25622 log.cc:824] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:55:15.541415 25622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d: No bootstrap required, opened a new log
I20260928 22:55:15.543921 25622 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4724083886345c99d3869d93bf07f4d" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46417 } }
I20260928 22:55:15.544090 25622 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:55:15.544121 25622 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4724083886345c99d3869d93bf07f4d, State: Initialized, Role: FOLLOWER
I20260928 22:55:15.544301 25622 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4724083886345c99d3869d93bf07f4d" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46417 } }
I20260928 22:55:15.544377 25622 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:55:15.544404 25622 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:55:15.544463 25622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:55:15.545106 25622 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4724083886345c99d3869d93bf07f4d" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46417 } }
I20260928 22:55:15.545210 25622 leader_election.cc:302] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4724083886345c99d3869d93bf07f4d; no voters:
I20260928 22:55:15.545404 25622 leader_election.cc:288] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:55:15.545465 25627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:55:15.545713 25627 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [term 1 LEADER]: Becoming Leader. State: Replica: c4724083886345c99d3869d93bf07f4d, State: Running, Role: LEADER
I20260928 22:55:15.545735 25622 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:55:15.545885 25627 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4724083886345c99d3869d93bf07f4d" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46417 } }
Sep 28 22:55:15 dist-test-slave-kzfr krb5kdc[25110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:55:15.547390 25622 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 22:55:15.547802 25628 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4724083886345c99d3869d93bf07f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4724083886345c99d3869d93bf07f4d" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46417 } } }
I20260928 22:55:15.549773 25629 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4724083886345c99d3869d93bf07f4d. Latest consensus state: current_term: 1 leader_uuid: "c4724083886345c99d3869d93bf07f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4724083886345c99d3869d93bf07f4d" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 46417 } } }
I20260928 22:55:15.549996 25629 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [sys.catalog]: This master's current role is: LEADER
I20260928 22:55:15.550499 25628 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d [sys.catalog]: This master's current role is: LEADER
I20260928 22:55:15.550848 25643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:55:15.551637 25643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:55:15.554476 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:15.540274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34680 (local address 127.14.14.190:46417)
0928 22:55:15.540470 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 22:55:15.540481 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:55:15.540574 (+ 93us) server_negotiation.cc:407] Connection header received
0928 22:55:15.541196 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:55:15.541203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:55:15.541256 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:55:15.541406 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:55:15.542066 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:15.543105 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:15.543885 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:15.544410 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:15.544512 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:15.547402 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:55:15.547406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:55:15.547407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:55:15.547419 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:55:15.552219 (+ 4800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:55:15.552698 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:55:15.552701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:55:15.552702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:55:15.552820 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:55:15.553093 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:55:15.553094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:55:15.553095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:55:15.553437 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:55:15.554004 (+ 567us) server_negotiation.cc:1035] Waiting for connection context
0928 22:55:15.554077 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 22:55:15.554238 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
I20260928 22:55:15.558079 25643 catalog_manager.cc:1347] Generated new cluster ID: ea26b6ff23934b1aac80d98495d39593
I20260928 22:55:15.558182 25643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:55:15.562077 25643 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:55:15.562845 25643 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:55:15.566903 25643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d: Generated new TSK 0
I20260928 22:55:15.567138 25643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:55:16.365728 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.14.148:52671/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46417
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:55:16.500447 25667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:55:16.500869 25667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:55:16.500967 25667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:55:16.503079 25667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:55:16.503157 25667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:55:16.503207 25667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:55:16.505841 25667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.14.148:52671/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:55:16.506392 25667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:55:16.506772 25667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:55:16.514158 25672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:55:16.514192 25675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:55:16.514161 25673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:55:16.514496 25667 server_base.cc:1034] running on GCE node
I20260928 22:55:16.515122 25667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:55:16.515405 25667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:55:16.516585 25667 hybrid_clock.cc:648] HybridClock initialized: now 1790636116516562 us; error 43 us; skew 500 ppm
Sep 28 22:55:16 dist-test-slave-kzfr krb5kdc[25110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636116, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:55:16.520061 25667 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:55:16.520596 25667 webserver.cc:460] Webserver started at http://127.14.14.129:43473/ using document root <none> and password file <none>
I20260928 22:55:16.520876 25667 fs_manager.cc:359] Metadata directory not provided
I20260928 22:55:16.521008 25667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:55:16.521214 25667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:55:16 dist-test-slave-kzfr krb5kdc[25110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636116, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
I20260928 22:55:16.540468 25667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "d5c900af92f94f6aa9ccac8570ebb8e0"
format_stamp: "Formatted at 2026-09-28 22:55:16 on dist-test-slave-kzfr"
server_key: "0be8e4a9369c494f945afb4f9b1cb582"
server_key_iv: "b6385b5da3d3cd305ecd7f8bf4a13367"
server_key_version: "kuduclusterkey@0"
I20260928 22:55:16.540952 25667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "d5c900af92f94f6aa9ccac8570ebb8e0"
format_stamp: "Formatted at 2026-09-28 22:55:16 on dist-test-slave-kzfr"
server_key: "0be8e4a9369c494f945afb4f9b1cb582"
server_key_iv: "b6385b5da3d3cd305ecd7f8bf4a13367"
server_key_version: "kuduclusterkey@0"
I20260928 22:55:16.543445 25667 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 22:55:16.561785 25682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:55:16.562072 25667 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260928 22:55:16.562247 25667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "d5c900af92f94f6aa9ccac8570ebb8e0"
format_stamp: "Formatted at 2026-09-28 22:55:16 on dist-test-slave-kzfr"
server_key: "0be8e4a9369c494f945afb4f9b1cb582"
server_key_iv: "b6385b5da3d3cd305ecd7f8bf4a13367"
server_key_version: "kuduclusterkey@0"
I20260928 22:55:16.562434 25667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:55:16.577529 25667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:55:16.583796 25667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:55:16.584040 25667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:55:16.584323 25667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:55:16.584794 25667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:55:16.584906 25667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:55:16.585032 25667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:55:16.585121 25667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:55:16.592357 25667 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:35111
I20260928 22:55:16.592927 25667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:55:16.595633 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 25667
I20260928 22:55:16.595747 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:55:16.595968 14394 external_mini_cluster.cc:1428] Setting key 21c2ce831cb66365be70d165b1369fa8
I20260928 22:55:16.605044 25797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:35111 every 8 connection(s)
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
Sep 28 22:55:16 dist-test-slave-kzfr krb5kdc[25110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636116, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:55:16.625309 25801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:16.605854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:36931 (local address 127.14.14.190:46417)
0928 22:55:16.605976 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 22:55:16.605979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:55:16.606165 (+ 186us) server_negotiation.cc:407] Connection header received
0928 22:55:16.606988 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:55:16.606989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:55:16.607004 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:55:16.607044 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:55:16.607936 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:16.608718 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:16.610346 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:16.610676 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:16.610722 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:16.621437 (+ 10715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:55:16.621440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:55:16.621440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:55:16.621450 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:55:16.623442 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:55:16.624049 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:55:16.624050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:55:16.624051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:55:16.624115 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:55:16.624410 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:55:16.624411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:55:16.624411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:55:16.624572 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:55:16.624617 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 22:55:16.625100 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 22:55:16.625203 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260928 22:55:16.626226 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:16.605857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:46417 (local address 127.14.14.129:36931)
0928 22:55:16.606041 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 22:55:16.606085 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:55:16.606853 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:55:16.607263 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:55:16.607268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:55:16.607530 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:55:16.607819 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:16.607825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:16.608839 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:16.608840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:16.610222 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:16.610244 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:16.610823 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:16.610824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:16.610963 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:16.615821 (+ 4858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:55:16.615832 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:55:16.621292 (+ 5460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:55:16.623558 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:55:16.623566 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:55:16.623570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:55:16.623955 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:55:16.624183 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:55:16.624184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:55:16.624184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:55:16.624329 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:55:16.624666 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:55:16.624673 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:55:16.624876 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 22:55:16.625679 (+ 803us) client_negotiation.cc:240] Negotiation successful
0928 22:55:16.625915 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260928 22:55:16.627003 25798 heartbeater.cc:347] Connected to a master server at 127.14.14.190:46417
I20260928 22:55:16.627099 25798 heartbeater.cc:464] Registering TS with master...
I20260928 22:55:16.627296 25798 heartbeater.cc:511] Master 127.14.14.190:46417 requested a full tablet report, sending...
I20260928 22:55:16.627810 25586 ts_manager.cc:194] Registered new tserver with Master: d5c900af92f94f6aa9ccac8570ebb8e0 (127.14.14.129:35111)
I20260928 22:55:16.628672 25586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:36931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:55:16.641323 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.14.148:52671/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46417
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:55:16.761193 25804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:55:16.761641 25804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:55:16.761754 25804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:55:16.763463 25804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:55:16.763676 25804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:55:16.763774 25804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:55:16.765933 25804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.14.148:52671/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:55:16.766451 25804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:55:16.766760 25804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:55:16.773134 25812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:55:16.773242 25811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:55:16.773128 25814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:55:16.774428 25804 server_base.cc:1034] running on GCE node
I20260928 22:55:16.774725 25804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:55:16.774966 25804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:55:16.776114 25804 hybrid_clock.cc:648] HybridClock initialized: now 1790636116776099 us; error 39 us; skew 500 ppm
Sep 28 22:55:16 dist-test-slave-kzfr krb5kdc[25110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636116, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:55:16.779256 25804 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:55:16.779731 25804 webserver.cc:460] Webserver started at http://127.14.14.130:34473/ using document root <none> and password file <none>
I20260928 22:55:16.779958 25804 fs_manager.cc:359] Metadata directory not provided
I20260928 22:55:16.780004 25804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:55:16.780122 25804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:55:16 dist-test-slave-kzfr krb5kdc[25110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636116, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
I20260928 22:55:16.794167 25804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f"
format_stamp: "Formatted at 2026-09-28 22:55:16 on dist-test-slave-kzfr"
server_key: "770a6042f681b2eecaf00f3c28e7b477"
server_key_iv: "a5bf10d1f7116898c5a2952db1724a41"
server_key_version: "kuduclusterkey@0"
I20260928 22:55:16.794627 25804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f"
format_stamp: "Formatted at 2026-09-28 22:55:16 on dist-test-slave-kzfr"
server_key: "770a6042f681b2eecaf00f3c28e7b477"
server_key_iv: "a5bf10d1f7116898c5a2952db1724a41"
server_key_version: "kuduclusterkey@0"
I20260928 22:55:16.797935 25804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
Sep 28 22:55:16 dist-test-slave-kzfr krb5kdc[25110](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790636116, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:55:16.817725 25821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:55:16.817924 25804 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 22:55:16.818040 25804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f"
format_stamp: "Formatted at 2026-09-28 22:55:16 on dist-test-slave-kzfr"
server_key: "770a6042f681b2eecaf00f3c28e7b477"
server_key_iv: "a5bf10d1f7116898c5a2952db1724a41"
server_key_version: "kuduclusterkey@0"
I20260928 22:55:16.818147 25804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:55:16.839465 25804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:55:16.842020 25804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:55:16.842247 25804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:55:16.842541 25804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:55:16.842983 25804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:55:16.843071 25804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:55:16.843145 25804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:55:16.843222 25804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:55:16.849637 25804 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:43119
I20260928 22:55:16.850102 25804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:55:16.859432 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 25804
I20260928 22:55:16.859539 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:55:16.859720 14394 external_mini_cluster.cc:1428] Setting key 5d204a68dcab98c4e0da251602cd9e5d
I20260928 22:55:16.862830 25934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:43119 every 8 connection(s)
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
Sep 28 22:55:16 dist-test-slave-kzfr krb5kdc[25110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636116, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:55:16.883070 25801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:16.863581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:52841 (local address 127.14.14.190:46417)
0928 22:55:16.863624 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:55:16.863626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:55:16.870914 (+ 7288us) server_negotiation.cc:407] Connection header received
0928 22:55:16.871624 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:55:16.871626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:55:16.871644 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:55:16.871674 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:55:16.872491 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:16.873317 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:16.876891 (+ 3574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:16.877215 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:16.877254 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:16.879797 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:55:16.879799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:55:16.879799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:55:16.879807 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:55:16.881483 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:55:16.882020 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:55:16.882021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:55:16.882022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:55:16.882083 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:55:16.882338 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:55:16.882339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:55:16.882340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:55:16.882468 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:55:16.882517 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:55:16.882877 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 22:55:16.882969 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:55:16.883827 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:16.863301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:46417 (local address 127.14.14.130:52841)
0928 22:55:16.870749 (+ 7448us) negotiation.cc:107] Waiting for socket to connect
0928 22:55:16.870813 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 22:55:16.871482 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:55:16.871884 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:55:16.871888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:55:16.872120 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:55:16.872390 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:16.872395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:16.873428 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:16.873429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:16.876784 (+ 3355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:16.876788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:16.877338 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:16.877339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:16.877477 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:16.878105 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:55:16.878116 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:55:16.879678 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:55:16.881617 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:55:16.881623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:55:16.881625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:55:16.881924 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:55:16.882149 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:55:16.882149 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:55:16.882150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:55:16.882268 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:55:16.882547 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:55:16.882552 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:55:16.882706 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 22:55:16.883419 (+ 713us) client_negotiation.cc:240] Negotiation successful
0928 22:55:16.883580 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7388,"thread_start_us":91,"threads_started":1}
I20260928 22:55:16.884491 25935 heartbeater.cc:347] Connected to a master server at 127.14.14.190:46417
I20260928 22:55:16.884598 25935 heartbeater.cc:464] Registering TS with master...
I20260928 22:55:16.884801 25935 heartbeater.cc:511] Master 127.14.14.190:46417 requested a full tablet report, sending...
I20260928 22:55:16.885176 25586 ts_manager.cc:194] Registered new tserver with Master: 5e3d5d09fbc340d3a3a1ee3a6afed50f (127.14.14.130:43119)
I20260928 22:55:16.885558 25586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:52841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:55:16.899466 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.14.148:52671/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46417
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:42849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:55:17.029964 25940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:55:17.030328 25940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:55:17.030421 25940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:55:17.032286 25940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:55:17.032537 25940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:55:17.032653 25940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:55:17.035131 25940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:42849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.14.148:52671/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:46417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:55:17.035630 25940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:55:17.035980 25940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:55:17.042894 25947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:55:17.043262 25946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:55:17.043503 25949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:55:17.044764 25940 server_base.cc:1034] running on GCE node
I20260928 22:55:17.045089 25940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:55:17.045392 25940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:55:17.046573 25940 hybrid_clock.cc:648] HybridClock initialized: now 1790636117046549 us; error 28 us; skew 500 ppm
Sep 28 22:55:17 dist-test-slave-kzfr krb5kdc[25110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636117, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:55:17.050455 25940 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:55:17.051008 25940 webserver.cc:460] Webserver started at http://127.14.14.131:36979/ using document root <none> and password file <none>
I20260928 22:55:17.051280 25940 fs_manager.cc:359] Metadata directory not provided
I20260928 22:55:17.051334 25940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:55:17.051465 25940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:55:17 dist-test-slave-kzfr krb5kdc[25110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636117, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
I20260928 22:55:17.069379 25940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "f63ea87a6e944fb786b0181d1cc15226"
format_stamp: "Formatted at 2026-09-28 22:55:17 on dist-test-slave-kzfr"
server_key: "603f2ac01baa2ad445ef6db64806828a"
server_key_iv: "9990afe8ddaf4f9d1432efbf64bace3a"
server_key_version: "kuduclusterkey@0"
I20260928 22:55:17.069941 25940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "f63ea87a6e944fb786b0181d1cc15226"
format_stamp: "Formatted at 2026-09-28 22:55:17 on dist-test-slave-kzfr"
server_key: "603f2ac01baa2ad445ef6db64806828a"
server_key_iv: "9990afe8ddaf4f9d1432efbf64bace3a"
server_key_version: "kuduclusterkey@0"
I20260928 22:55:17.086040 25940 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.002s sys 0.000s
I20260928 22:55:17.106730 25956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:55:17.109212 25940 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260928 22:55:17.109334 25940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "f63ea87a6e944fb786b0181d1cc15226"
format_stamp: "Formatted at 2026-09-28 22:55:17 on dist-test-slave-kzfr"
server_key: "603f2ac01baa2ad445ef6db64806828a"
server_key_iv: "9990afe8ddaf4f9d1432efbf64bace3a"
server_key_version: "kuduclusterkey@0"
I20260928 22:55:17.109439 25940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:55:17.134862 25940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:55:17.144039 25940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:55:17.144289 25940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:55:17.144595 25940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:55:17.145000 25940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:55:17.145057 25940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:55:17.145135 25940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:55:17.145167 25940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:55:17.152709 25940 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:46113
I20260928 22:55:17.153103 25940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:55:17.153775 26069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:46113 every 8 connection(s)
I20260928 22:55:17.159888 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 25940
I20260928 22:55:17.159994 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:55:17.160173 14394 external_mini_cluster.cc:1428] Setting key 4a1500ea318000fe6fc5479c622ca8a0
Sep 28 22:55:17 dist-test-slave-kzfr krb5kdc[25110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636117, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:55:17.168284 25801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:17.157278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:48289 (local address 127.14.14.190:46417)
0928 22:55:17.157339 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:55:17.157342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:55:17.157695 (+ 353us) server_negotiation.cc:407] Connection header received
0928 22:55:17.158700 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:55:17.158702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:55:17.158715 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:55:17.158749 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:55:17.159614 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.160392 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.161934 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.162239 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.162286 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:17.164925 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:55:17.164927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:55:17.164928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:55:17.164935 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:55:17.166747 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:55:17.167217 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:55:17.167219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:55:17.167219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:55:17.167288 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:55:17.167558 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:55:17.167558 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:55:17.167559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:55:17.167688 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:55:17.167730 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 22:55:17.168087 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 22:55:17.168173 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:55:17.169023 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:17.157295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:46417 (local address 127.14.14.131:48289)
0928 22:55:17.157552 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 22:55:17.157611 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 22:55:17.158565 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:55:17.158934 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:55:17.158938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:55:17.159184 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:55:17.159492 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:17.159500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.160513 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.160514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:17.161834 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:17.161836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.162379 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.162380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:17.162518 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:17.163114 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:55:17.163124 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:55:17.164816 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:55:17.166843 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:55:17.166849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:55:17.166852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:55:17.167129 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:55:17.167360 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:55:17.167361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:55:17.167361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:55:17.167487 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:55:17.167777 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:55:17.167783 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:55:17.167936 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 22:55:17.168633 (+ 697us) client_negotiation.cc:240] Negotiation successful
0928 22:55:17.168794 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20260928 22:55:17.169708 26070 heartbeater.cc:347] Connected to a master server at 127.14.14.190:46417
I20260928 22:55:17.169798 26070 heartbeater.cc:464] Registering TS with master...
I20260928 22:55:17.169972 26070 heartbeater.cc:511] Master 127.14.14.190:46417 requested a full tablet report, sending...
I20260928 22:55:17.170408 25586 ts_manager.cc:194] Registered new tserver with Master: f63ea87a6e944fb786b0181d1cc15226 (127.14.14.131:46113)
I20260928 22:55:17.170845 25586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:48289
I20260928 22:55:17.171662 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:55:17.188161 25801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:17.179980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.14.14.190:46417)
0928 22:55:17.180033 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:55:17.180036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:55:17.180252 (+ 216us) server_negotiation.cc:407] Connection header received
0928 22:55:17.180365 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:55:17.180366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:55:17.180377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:55:17.180410 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:55:17.181264 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.182092 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.183230 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.183528 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.183569 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:17.185155 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:55:17.185156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:55:17.185156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:55:17.185163 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:55:17.186906 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:55:17.187336 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:55:17.187337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:55:17.187338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:55:17.187394 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:55:17.187584 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:55:17.187585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:55:17.187585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:55:17.187702 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:55:17.187783 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:55:17.187990 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 22:55:17.188065 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:55:17.190436 25586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:55:17.192768 25586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:55:17.207044 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:17.196385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:46113 (local address 127.0.0.1:53966)
0928 22:55:17.201939 (+ 5554us) negotiation.cc:107] Waiting for socket to connect
0928 22:55:17.201993 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 22:55:17.202082 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:55:17.202431 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:55:17.202435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:55:17.202438 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:55:17.202516 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:17.202518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.203855 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.203856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:17.205617 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:17.205620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.206718 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.206719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:17.206813 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:17.206828 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 22:55:17.206904 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 22:55:17.206955 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5520,"mutex_wait_us":70,"thread_start_us":72,"threads_started":1}
I20260928 22:55:17.207645 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:17.197148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.14.14.131:46113)
0928 22:55:17.201698 (+ 4550us) server_negotiation.cc:206] Beginning negotiation
0928 22:55:17.201704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:55:17.202095 (+ 391us) server_negotiation.cc:407] Connection header received
0928 22:55:17.202193 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:55:17.202196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:55:17.202215 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:55:17.202310 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:55:17.202607 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.203748 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.205752 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.206525 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.206602 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:17.206681 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:55:17.207516 (+ 835us) server_negotiation.cc:299] Negotiation successful
0928 22:55:17.207558 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4503,"thread_start_us":98,"threads_started":1}
I20260928 22:55:17.208873 26004 tablet_service.cc:1467] Processing CreateTablet for tablet 8dbe6df6442c45a8a97d2ad9adcd7848 (DEFAULT_TABLE table=test-table [id=55e6e1702dd44594a1d4d30c39bf904a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:55:17.209261 26004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dbe6df6442c45a8a97d2ad9adcd7848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:55:17.213096 26088 tablet_bootstrap.cc:492] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226: Bootstrap starting.
I20260928 22:55:17.214430 26088 tablet_bootstrap.cc:654] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226: Neither blocks nor log segments found. Creating new log.
I20260928 22:55:17.214775 26088 log.cc:824] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226: Log is configured to *not* fsync() on all Append() calls
I20260928 22:55:17.215026 26082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:17.196620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:43119 (local address 127.0.0.1:34208)
0928 22:55:17.202210 (+ 5590us) negotiation.cc:107] Waiting for socket to connect
0928 22:55:17.202219 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:55:17.202257 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:55:17.202839 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:55:17.202839 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:55:17.202841 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:55:17.202883 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:17.202885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.204047 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.204047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:17.205269 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:17.205272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.214755 (+ 9483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.214757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:17.214836 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:17.214841 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:55:17.214900 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:55:17.214947 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5576,"thread_start_us":69,"threads_started":1}
I20260928 22:55:17.215306 26087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:17.197335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34208 (local address 127.14.14.130:43119)
0928 22:55:17.201790 (+ 4455us) server_negotiation.cc:206] Beginning negotiation
0928 22:55:17.201793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:55:17.202314 (+ 521us) server_negotiation.cc:407] Connection header received
0928 22:55:17.202362 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:55:17.202365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:55:17.202380 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:55:17.202443 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:55:17.202956 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.203924 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.205393 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.207509 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.207576 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:17.207640 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:55:17.215173 (+ 7533us) server_negotiation.cc:299] Negotiation successful
0928 22:55:17.215229 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4420,"thread_start_us":73,"threads_started":1}
I20260928 22:55:17.215803 26083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:17.196821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:35111 (local address 127.0.0.1:41260)
0928 22:55:17.202543 (+ 5722us) negotiation.cc:107] Waiting for socket to connect
0928 22:55:17.202552 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:55:17.202593 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:55:17.204021 (+ 1428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:55:17.204022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:55:17.204024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:55:17.205668 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:17.205671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.207672 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.207673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:17.209604 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:17.209607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.215568 (+ 5961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.215569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:17.215642 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:17.215647 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:55:17.215694 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 22:55:17.215737 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5708,"mutex_wait_us":1587,"thread_start_us":56,"threads_started":1}
I20260928 22:55:17.216043 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:17.196991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41260 (local address 127.14.14.129:35111)
0928 22:55:17.201867 (+ 4876us) server_negotiation.cc:206] Beginning negotiation
0928 22:55:17.201871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:55:17.202650 (+ 779us) server_negotiation.cc:407] Connection header received
0928 22:55:17.202689 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:55:17.202691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:55:17.202707 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:55:17.202771 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:55:17.205765 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.206860 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.209703 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.210460 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.210529 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:17.210576 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:55:17.215925 (+ 5349us) server_negotiation.cc:299] Negotiation successful
0928 22:55:17.215980 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4842,"thread_start_us":70,"threads_started":1}
I20260928 22:55:17.216012 25858 tablet_service.cc:1467] Processing CreateTablet for tablet 8dbe6df6442c45a8a97d2ad9adcd7848 (DEFAULT_TABLE table=test-table [id=55e6e1702dd44594a1d4d30c39bf904a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:55:17.216267 26088 tablet_bootstrap.cc:492] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226: No bootstrap required, opened a new log
I20260928 22:55:17.216358 25858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dbe6df6442c45a8a97d2ad9adcd7848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:55:17.216375 26088 ts_tablet_manager.cc:1397] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 22:55:17.218736 26088 raft_consensus.cc:348] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63ea87a6e944fb786b0181d1cc15226" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 46113 } } peers { permanent_uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43119 } } peers { permanent_uuid: "d5c900af92f94f6aa9ccac8570ebb8e0" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35111 } }
I20260928 22:55:17.219033 26088 raft_consensus.cc:374] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:55:17.219129 26088 raft_consensus.cc:729] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f63ea87a6e944fb786b0181d1cc15226, State: Initialized, Role: FOLLOWER
I20260928 22:55:17.219352 26088 consensus_queue.cc:260] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63ea87a6e944fb786b0181d1cc15226" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 46113 } } peers { permanent_uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43119 } } peers { permanent_uuid: "d5c900af92f94f6aa9ccac8570ebb8e0" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35111 } }
I20260928 22:55:17.219743 26088 ts_tablet_manager.cc:1428] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:55:17.219913 26070 heartbeater.cc:503] Master 127.14.14.190:46417 was elected leader, sending a full tablet report...
I20260928 22:55:17.221035 25732 tablet_service.cc:1467] Processing CreateTablet for tablet 8dbe6df6442c45a8a97d2ad9adcd7848 (DEFAULT_TABLE table=test-table [id=55e6e1702dd44594a1d4d30c39bf904a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:55:17.221345 26091 tablet_bootstrap.cc:492] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f: Bootstrap starting.
I20260928 22:55:17.221446 25732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dbe6df6442c45a8a97d2ad9adcd7848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:55:17.222386 26091 tablet_bootstrap.cc:654] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f: Neither blocks nor log segments found. Creating new log.
I20260928 22:55:17.222762 26091 log.cc:824] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:55:17.224035 26091 tablet_bootstrap.cc:492] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f: No bootstrap required, opened a new log
I20260928 22:55:17.224095 26091 ts_tablet_manager.cc:1397] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 22:55:17.226339 26091 raft_consensus.cc:348] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63ea87a6e944fb786b0181d1cc15226" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 46113 } } peers { permanent_uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43119 } } peers { permanent_uuid: "d5c900af92f94f6aa9ccac8570ebb8e0" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35111 } }
I20260928 22:55:17.226588 26091 raft_consensus.cc:374] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:55:17.226680 26091 raft_consensus.cc:729] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e3d5d09fbc340d3a3a1ee3a6afed50f, State: Initialized, Role: FOLLOWER
I20260928 22:55:17.226886 26091 consensus_queue.cc:260] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63ea87a6e944fb786b0181d1cc15226" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 46113 } } peers { permanent_uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43119 } } peers { permanent_uuid: "d5c900af92f94f6aa9ccac8570ebb8e0" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35111 } }
I20260928 22:55:17.227660 26091 ts_tablet_manager.cc:1428] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 22:55:17.227831 25935 heartbeater.cc:503] Master 127.14.14.190:46417 was elected leader, sending a full tablet report...
I20260928 22:55:17.229789 26093 tablet_bootstrap.cc:492] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0: Bootstrap starting.
I20260928 22:55:17.230645 26093 tablet_bootstrap.cc:654] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0: Neither blocks nor log segments found. Creating new log.
I20260928 22:55:17.231082 26093 log.cc:824] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0: Log is configured to *not* fsync() on all Append() calls
I20260928 22:55:17.242434 26093 tablet_bootstrap.cc:492] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0: No bootstrap required, opened a new log
I20260928 22:55:17.242625 26093 ts_tablet_manager.cc:1397] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260928 22:55:17.246049 26093 raft_consensus.cc:348] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63ea87a6e944fb786b0181d1cc15226" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 46113 } } peers { permanent_uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43119 } } peers { permanent_uuid: "d5c900af92f94f6aa9ccac8570ebb8e0" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35111 } }
I20260928 22:55:17.246393 26093 raft_consensus.cc:374] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:55:17.246485 26093 raft_consensus.cc:729] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c900af92f94f6aa9ccac8570ebb8e0, State: Initialized, Role: FOLLOWER
I20260928 22:55:17.246698 26093 consensus_queue.cc:260] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63ea87a6e944fb786b0181d1cc15226" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 46113 } } peers { permanent_uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43119 } } peers { permanent_uuid: "d5c900af92f94f6aa9ccac8570ebb8e0" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35111 } }
I20260928 22:55:17.247134 26093 ts_tablet_manager.cc:1428] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 22:55:17.247323 25798 heartbeater.cc:503] Master 127.14.14.190:46417 was elected leader, sending a full tablet report...
Sep 28 22:55:17 dist-test-slave-kzfr krb5kdc[25110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636116, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
W20260928 22:55:17.355890 25799 tablet.cc:2369] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:55:17.363492 25936 tablet.cc:2369] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:55:17.404299 26071 tablet.cc:2369] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:55:17.426339 26096 raft_consensus.cc:482] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:55:17.426496 26096 raft_consensus.cc:504] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63ea87a6e944fb786b0181d1cc15226" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 46113 } } peers { permanent_uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43119 } } peers { permanent_uuid: "d5c900af92f94f6aa9ccac8570ebb8e0" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35111 } }
I20260928 22:55:17.426889 26096 leader_election.cc:288] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f63ea87a6e944fb786b0181d1cc15226 (127.14.14.131:46113), 5e3d5d09fbc340d3a3a1ee3a6afed50f (127.14.14.130:43119)
I20260928 22:55:17.431475 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:17.427502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:40959 (local address 127.14.14.131:46113)
0928 22:55:17.427552 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:55:17.427554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:55:17.427767 (+ 213us) server_negotiation.cc:407] Connection header received
0928 22:55:17.427869 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:55:17.427870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:55:17.427879 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:55:17.427907 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:55:17.428155 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.428905 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.430583 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.431100 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.431146 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:17.431154 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:55:17.431362 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 22:55:17.431402 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:55:17.431932 26098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:17.427093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:46113 (local address 127.14.14.129:40959)
0928 22:55:17.427694 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0928 22:55:17.427705 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:55:17.427796 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:55:17.427973 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:55:17.427973 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:55:17.427987 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:55:17.428073 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:17.428075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.429034 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.429035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:17.430480 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:17.430483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.431212 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.431213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:17.431281 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:17.431284 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:55:17.431799 (+ 515us) client_negotiation.cc:240] Negotiation successful
0928 22:55:17.431846 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":87,"threads_started":1}
I20260928 22:55:17.432397 26024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbe6df6442c45a8a97d2ad9adcd7848" candidate_uuid: "d5c900af92f94f6aa9ccac8570ebb8e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f63ea87a6e944fb786b0181d1cc15226" is_pre_election: true
I20260928 22:55:17.432556 26024 raft_consensus.cc:2454] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c900af92f94f6aa9ccac8570ebb8e0 in term 0.
I20260928 22:55:17.432891 25685 leader_election.cc:302] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5c900af92f94f6aa9ccac8570ebb8e0, f63ea87a6e944fb786b0181d1cc15226; no voters:
I20260928 22:55:17.433025 26096 raft_consensus.cc:2781] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:55:17.433063 26096 raft_consensus.cc:482] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:55:17.433082 26096 raft_consensus.cc:3037] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:55:17.434136 26096 raft_consensus.cc:504] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63ea87a6e944fb786b0181d1cc15226" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 46113 } } peers { permanent_uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43119 } } peers { permanent_uuid: "d5c900af92f94f6aa9ccac8570ebb8e0" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35111 } }
I20260928 22:55:17.434268 26096 leader_election.cc:288] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [CANDIDATE]: Term 1 election: Requested vote from peers f63ea87a6e944fb786b0181d1cc15226 (127.14.14.131:46113), 5e3d5d09fbc340d3a3a1ee3a6afed50f (127.14.14.130:43119)
I20260928 22:55:17.434536 26024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbe6df6442c45a8a97d2ad9adcd7848" candidate_uuid: "d5c900af92f94f6aa9ccac8570ebb8e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f63ea87a6e944fb786b0181d1cc15226"
I20260928 22:55:17.434609 26024 raft_consensus.cc:3037] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:55:17.435412 26024 raft_consensus.cc:2454] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c900af92f94f6aa9ccac8570ebb8e0 in term 1.
I20260928 22:55:17.435624 25685 leader_election.cc:302] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5c900af92f94f6aa9ccac8570ebb8e0, f63ea87a6e944fb786b0181d1cc15226; no voters:
I20260928 22:55:17.435729 26096 raft_consensus.cc:2781] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:55:17.435878 26096 raft_consensus.cc:686] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [term 1 LEADER]: Becoming Leader. State: Replica: d5c900af92f94f6aa9ccac8570ebb8e0, State: Running, Role: LEADER
I20260928 22:55:17.435961 26096 consensus_queue.cc:237] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63ea87a6e944fb786b0181d1cc15226" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 46113 } } peers { permanent_uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43119 } } peers { permanent_uuid: "d5c900af92f94f6aa9ccac8570ebb8e0" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35111 } }
I20260928 22:55:17.436733 25586 catalog_manager.cc:5495] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5c900af92f94f6aa9ccac8570ebb8e0 (127.14.14.129). New cstate: current_term: 1 leader_uuid: "d5c900af92f94f6aa9ccac8570ebb8e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63ea87a6e944fb786b0181d1cc15226" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 46113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c900af92f94f6aa9ccac8570ebb8e0" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35111 } health_report { overall_health: HEALTHY } } }
I20260928 22:55:17.447232 26087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:17.427560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:44017 (local address 127.14.14.130:43119)
0928 22:55:17.427617 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:55:17.427619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:55:17.429441 (+ 1822us) server_negotiation.cc:407] Connection header received
0928 22:55:17.429485 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:55:17.429486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:55:17.429500 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:55:17.429569 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:55:17.438242 (+ 8673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.438987 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.446076 (+ 7089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.446715 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.446780 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:17.446791 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:55:17.447111 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 22:55:17.447160 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:55:17.447257 26097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:17.427295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:43119 (local address 127.14.14.129:44017)
0928 22:55:17.429372 (+ 2077us) negotiation.cc:107] Waiting for socket to connect
0928 22:55:17.429383 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:55:17.429441 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:55:17.438090 (+ 8649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:55:17.438091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:55:17.438093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:55:17.438154 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:17.438156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.439087 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.439088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:17.445920 (+ 6832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:55:17.445926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.446934 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.446935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:55:17.447019 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:17.447023 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:55:17.447083 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:55:17.447128 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2058,"thread_start_us":73,"threads_started":1}
I20260928 22:55:17.447867 25888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbe6df6442c45a8a97d2ad9adcd7848" candidate_uuid: "d5c900af92f94f6aa9ccac8570ebb8e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f"
I20260928 22:55:17.447876 25889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbe6df6442c45a8a97d2ad9adcd7848" candidate_uuid: "d5c900af92f94f6aa9ccac8570ebb8e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f" is_pre_election: true
I20260928 22:55:17.448011 25888 raft_consensus.cc:3037] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:55:17.449018 25888 raft_consensus.cc:2454] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c900af92f94f6aa9ccac8570ebb8e0 in term 1.
I20260928 22:55:17.488137 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:17.484796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.14.14.129:35111)
0928 22:55:17.484876 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:55:17.484878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:55:17.484888 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:55:17.484935 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:55:17.484936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:55:17.484958 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:55:17.485023 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:55:17.485198 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.485994 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.487053 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:55:17.487327 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:55:17.487381 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:55:17.487561 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:55:17.487699 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:55:17.487780 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:55:17.488013 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 22:55:17.488034 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 22:55:17.490370 25888 raft_consensus.cc:1261] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f [term 1 FOLLOWER]: Refusing update from remote peer d5c900af92f94f6aa9ccac8570ebb8e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:55:17.490430 26024 raft_consensus.cc:1261] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226 [term 1 FOLLOWER]: Refusing update from remote peer d5c900af92f94f6aa9ccac8570ebb8e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:55:17.490772 26099 consensus_queue.cc:1035] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e3d5d09fbc340d3a3a1ee3a6afed50f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:55:17.490887 26096 consensus_queue.cc:1035] T 8dbe6df6442c45a8a97d2ad9adcd7848 P d5c900af92f94f6aa9ccac8570ebb8e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f63ea87a6e944fb786b0181d1cc15226" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 46113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:55:17.495277 26102 mvcc.cc:205] Tried to move back new op lower bound from 7334445537237307392 to 7334445537018122240. Current Snapshot: MvccSnapshot[applied={T|T < 7334445537237307392}]
I20260928 22:55:17.495440 26104 mvcc.cc:205] Tried to move back new op lower bound from 7334445537237307392 to 7334445537018122240. Current Snapshot: MvccSnapshot[applied={T|T < 7334445537237307392}]
I20260928 22:55:17.499166 25585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:55:17.499280 25585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:55:17.500321 25585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4724083886345c99d3869d93bf07f4d: Sending DeleteTablet for 3 replicas of tablet 8dbe6df6442c45a8a97d2ad9adcd7848
I20260928 22:55:17.500779 26004 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dbe6df6442c45a8a97d2ad9adcd7848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:55:17 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:53966
I20260928 22:55:17.500985 25858 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dbe6df6442c45a8a97d2ad9adcd7848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:55:17 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:34208
I20260928 22:55:17.501436 25732 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dbe6df6442c45a8a97d2ad9adcd7848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:55:17 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:41260
I20260928 22:55:17.502040 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 25667
I20260928 22:55:17.503819 26112 tablet_replica.cc:330] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f: stopping tablet replica
I20260928 22:55:17.503959 26111 tablet_replica.cc:330] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226: stopping tablet replica
I20260928 22:55:17.504117 26111 raft_consensus.cc:2229] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:55:17.504276 26112 raft_consensus.cc:2229] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:55:17.504433 26111 raft_consensus.cc:2258] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:55:17.504570 26112 raft_consensus.cc:2258] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:55:17.505247 26112 ts_tablet_manager.cc:1905] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:55:17.505551 26111 ts_tablet_manager.cc:1905] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:55:17.508642 26112 ts_tablet_manager.cc:1918] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:55:17.508826 26111 ts_tablet_manager.cc:1918] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:55:17.508929 26111 log.cc:1196] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/8dbe6df6442c45a8a97d2ad9adcd7848
I20260928 22:55:17.509025 26112 log.cc:1196] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/8dbe6df6442c45a8a97d2ad9adcd7848
I20260928 22:55:17.509667 26111 ts_tablet_manager.cc:1939] T 8dbe6df6442c45a8a97d2ad9adcd7848 P f63ea87a6e944fb786b0181d1cc15226: Deleting consensus metadata
I20260928 22:55:17.509967 26112 ts_tablet_manager.cc:1939] T 8dbe6df6442c45a8a97d2ad9adcd7848 P 5e3d5d09fbc340d3a3a1ee3a6afed50f: Deleting consensus metadata
I20260928 22:55:17.510478 25570 catalog_manager.cc:4841] TS 5e3d5d09fbc340d3a3a1ee3a6afed50f (127.14.14.130:43119): tablet 8dbe6df6442c45a8a97d2ad9adcd7848 (table test-table [id=55e6e1702dd44594a1d4d30c39bf904a]) successfully deleted
I20260928 22:55:17.511039 25572 catalog_manager.cc:4841] TS f63ea87a6e944fb786b0181d1cc15226 (127.14.14.131:46113): tablet 8dbe6df6442c45a8a97d2ad9adcd7848 (table test-table [id=55e6e1702dd44594a1d4d30c39bf904a]) successfully deleted
W20260928 22:55:17.515300 25570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:35111 (error 108)
I20260928 22:55:17.515982 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 25804
I20260928 22:55:17.521952 26083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:55:17.521814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:35111 (local address 127.0.0.1:41286)
0928 22:55:17.521861 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 22:55:17.521912 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:35111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 22:55:17.521950 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 25940
W20260928 22:55:17.522526 25570 catalog_manager.cc:4584] TS d5c900af92f94f6aa9ccac8570ebb8e0 (127.14.14.129:35111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8dbe6df6442c45a8a97d2ad9adcd7848: Network error: Client connection negotiation failed: client connection to 127.14.14.129:35111: connect: Connection refused (error 111)
I20260928 22:55:17.527619 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 25554
I20260928 22:55:17.533844 14394 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 22:55:17.878473 14394 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 22:55:17.878589 14394 mini_ranger.cc:66] Stopping Ranger...
I20260928 22:55:18.223609 14394 mini_ranger.cc:68] Stopped Ranger
2026-09-28 22:55:18.223 UTC [25140] LOG: received smart shutdown request
2026-09-28 22:55:18.227 UTC [25140] LOG: background worker "logical replication launcher" (PID 25149) exited with exit code 1
2026-09-28 22:55:18.227 UTC [25144] LOG: shutting down
2026-09-28 22:55:18.410 UTC [25140] LOG: database system is shut down
2026-09-28T22:55:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47652 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:55:18 dist-test-slave-kzfr krb5kdc[26123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:55:18 dist-test-slave-kzfr krb5kdc[26123](info): set up 2 sockets
Sep 28 22:55:18 dist-test-slave-kzfr krb5kdc[26123](info): commencing operation
krb5kdc: starting...
W20260928 22:55:19.605769 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:55:19 dist-test-slave-kzfr krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:55:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:55:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.14.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.14.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/rangeradmin_127.14.14.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/rangeradmin_127.14.14.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.14.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.14.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/rangerlookup_127.14.14.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/rangerlookup_127.14.14.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.14.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.14.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/HTTP_127.14.14.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/HTTP_127.14.14.148@KRBTEST.COM.keytab.
I20260928 22:55:19.778584 14394 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkdZKSl/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/postgres -l logfile start
2026-09-28 22:55:22.447 UTC [26153] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 22:55:22.447 UTC [26153] LOG: listening on IPv4 address "127.14.14.148", port 48921
2026-09-28 22:55:22.452 UTC [26153] LOG: listening on Unix socket "/tmp/.s.PGSQL.48921"
2026-09-28 22:55:22.481 UTC [26156] LOG: database system was shut down at 2026-09-28 22:55:20 UTC
2026-09-28 22:55:22.485 UTC [26153] LOG: database system is ready to accept connections
I20260928 22:55:23.437687 14394 mini_postgres.cc:96] Postgres bound to 48921
2026-09-28 22:55:23.443 UTC [26164] FATAL: database "slave" does not exist
127.14.14.148:48921 - accepting connections
I20260928 22:55:23.444568 14394 mini_ranger.cc:161] Starting Ranger...
I20260928 22:55:23.459599 14394 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 22:55:24.001356 14394 mini_ranger.cc:87] Created ranger Postgres database
I20260928 22:55:24.001446 14394 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-admin
2026-09-28 22:55:24,269 [I] DB FLAVOR :POSTGRES
2026-09-28 22:55:24,270 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 22:55:24,270 [I] Checking connection..
2026-09-28 22:55:24,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 22:55:24,527 [I] Checking connection passed.
2026-09-28 22:55:24,528 [I] --------- Verifying version history table ---------
2026-09-28 22:55:24,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:55:24,783 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 22:55:24,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:55:25,048 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 22:55:25,048 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 22:55:25,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 22:55:25.277 UTC [26228] WARNING: there is no transaction in progress
2026-09-28 22:55:25,284 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 22:55:25,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:55:25,549 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 22:55:25,549 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 22:55:25,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 22:55:25,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 22:55:26,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kzfr.c.gcp-upstream.internal','N') ;"
2026-09-28 22:55:26,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:55:26,528 [I] Table x_portal_user does not exist in database ranger
2026-09-28 22:55:26,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:55:26,780 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 22:55:26,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:55:27,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:55:27,251 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 22:55:27,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 22:55:28.238 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:28.255 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:28.272 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:28.289 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:28.305 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:28.323 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:28.379 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:28.386 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:28.404 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:28.415 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:28.425 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:28.438 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:28.448 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:28.455 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:28.462 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:28.470 UTC [26336] WARNING: there is no transaction in progress
2026-09-28 22:55:29,164 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 22:55:29,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 22:55:29,406 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 22:55:29,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:55:29,640 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 22:55:29,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:55:29,874 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 22:55:29,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:55:30,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:55:30,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kzfr.c.gcp-upstream.internal';"
2026-09-28 22:55:30,611 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 22:55:30,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 22:55:30,836 [I] DB_PATCHES have already been applied
I20260928 22:55:30.841981 14394 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkdZKSl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 22:55:30.842062 14394 mini_ranger.cc:193] Using host: 127.14.14.148
I20260928 22:55:30.844973 14394 mini_ranger.cc:227] Ranger admin URL: http://127.14.14.148:36549
Sep 28, 2026 10:55:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:55:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:55:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 10:55:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 10:55:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 10:55:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 10:55:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 22:55:32.020470 14394 mini_ranger.cc:160] Time spent starting Ranger: real 8.576s user 0.000s sys 0.009s
Sep 28, 2026 10:55:50 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 10:55:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 10:55:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 10:55:50 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 10:55:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 10:55:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 10:55:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 10:55:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 10:55:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 10:55:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 10:55:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 10:55:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 10:55:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 10:55:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 10:55:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 10:55:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 10:55:50 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 10:55:51 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 22:55:52.481181 14394 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.14.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.14.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/rangerkms_127.14.14.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.14.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/rangerkms_127.14.14.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.14.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.14.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.14.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/HTTP_127.14.14.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.14.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/HTTP_127.14.14.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 22:55:52 dist-test-slave-kzfr krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636152, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 22:55:52.566876 14394 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 22:55:52.576239 14394 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 22:55:53.560514 14394 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 22:55:53.560601 14394 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-kms
I20260928 22:55:53.560619 14394 mini_ranger_kms.cc:227] Using postgres at 127.14.14.148:48921
2026-09-28 22:55:53,740 [I] DB FLAVOR :POSTGRES
2026-09-28 22:55:53,740 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 22:55:53,740 [I] Checking connection
2026-09-28 22:55:53,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 22:55:54,010 [I] connection success
2026-09-28 22:55:54,010 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 22:55:54,010 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 22:55:54,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 22:55:54,260 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 22:55:54,260 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 22:55:54,260 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 22:55:54,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.14.148:48921/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 22:55:54,544 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 22:55:55.037385 14394 mini_ranger_kms.cc:310] Created kms service
I20260928 22:55:55.232235 14394 mini_ranger_kms.cc:326] Created kudu user
I20260928 22:55:55.276455 14394 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 22:55:55 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636152, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
I20260928 22:55:55.691957 14394 mini_ranger_kms.cc:384] Added ranger policy
I20260928 22:55:55.692181 14394 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkdZKSl/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkdZKSl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 22:55:55.692209 14394 mini_ranger_kms.cc:240] Using host: 127.14.14.148
I20260928 22:55:55.696180 14394 mini_ranger_kms.cc:276] Ranger KMS PID: 26516
I20260928 22:55:55.696264 14394 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.14.148:58057
Sep 28, 2026 10:55:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:55:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:55:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 10:55:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 10:55:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 10:55:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 22:55:58.074501 14394 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.508s user 0.018s sys 0.144s
I20260928 22:55:58.074612 14394 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 22:55:58.074659 14394 mini_ranger_kms.cc:401] 127.14.14.148:58057/kms/v1/keys
Sep 28 22:55:58 dist-test-slave-kzfr krb5kdc[26123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790636158, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.14.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 22:55:58 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636158, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.14.148@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
Sep 28 22:55:59 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636158, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.14.148@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
Sep 28 22:56:01 dist-test-slave-kzfr krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:56:01.302213 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:38957
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:38957
--ranger_config_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.14.148:58057/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:01.384618 26567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:01.384845 26567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:01.384866 26567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:01.386098 26567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:56:01.386140 26567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:01.386155 26567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:01.386168 26567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:56:01.386180 26567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:56:01.387862 26567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.14.148:58057/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:38957
--ranger_config_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:01.388104 26567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:01.388352 26567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:01.393406 26572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:01.393317 26573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:01.393335 26575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:01.393805 26567 server_base.cc:1034] running on GCE node
I20260928 22:56:01.394037 26567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:01.394320 26567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:01.395494 26567 hybrid_clock.cc:648] HybridClock initialized: now 1790636161395480 us; error 31 us; skew 500 ppm
Sep 28 22:56:01 dist-test-slave-kzfr krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636161, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:01.401377 26567 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:56:01.401937 26567 webserver.cc:460] Webserver started at http://127.14.14.190:38019/ using document root <none> and password file <none>
I20260928 22:56:01.402168 26567 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:01.402225 26567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:01.402369 26567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:56:01 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636161, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
I20260928 22:56:01.466492 26567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "b069d890279d4821ac97cf265e3a59bc"
format_stamp: "Formatted at 2026-09-28 22:56:01 on dist-test-slave-kzfr"
server_key: "9f4f486e299c99dcc797a36adb8fe7cb"
server_key_iv: "c39721f7c1b7bd5f14b951e90e31829a"
server_key_version: "kuduclusterkey@0"
I20260928 22:56:01.466956 26567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "b069d890279d4821ac97cf265e3a59bc"
format_stamp: "Formatted at 2026-09-28 22:56:01 on dist-test-slave-kzfr"
server_key: "9f4f486e299c99dcc797a36adb8fe7cb"
server_key_iv: "c39721f7c1b7bd5f14b951e90e31829a"
server_key_version: "kuduclusterkey@0"
I20260928 22:56:01.468498 26567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:56:01.485780 26582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:01.486047 26567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:56:01.486187 26567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "b069d890279d4821ac97cf265e3a59bc"
format_stamp: "Formatted at 2026-09-28 22:56:01 on dist-test-slave-kzfr"
server_key: "9f4f486e299c99dcc797a36adb8fe7cb"
server_key_iv: "c39721f7c1b7bd5f14b951e90e31829a"
server_key_version: "kuduclusterkey@0"
I20260928 22:56:01.486299 26567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:01.505373 26567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:01.507743 26567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:01.507920 26567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:01.513944 26567 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:38957
I20260928 22:56:01.513991 26634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:38957 every 8 connection(s)
I20260928 22:56:01.514458 26567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:56:01.515187 26635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:01.517416 26635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc: Bootstrap starting.
I20260928 22:56:01.518066 26635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:01.518193 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 26567
I20260928 22:56:01.518308 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:56:01.518431 26635 log.cc:824] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:01.518497 14394 external_mini_cluster.cc:1428] Setting key b565624403b6b3f6edbd8940f1a5cde1
I20260928 22:56:01.519474 26635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc: No bootstrap required, opened a new log
I20260928 22:56:01.521808 26635 raft_consensus.cc:348] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b069d890279d4821ac97cf265e3a59bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38957 } }
I20260928 22:56:01.521973 26635 raft_consensus.cc:374] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:01.522002 26635 raft_consensus.cc:729] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b069d890279d4821ac97cf265e3a59bc, State: Initialized, Role: FOLLOWER
I20260928 22:56:01.522166 26635 consensus_queue.cc:260] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b069d890279d4821ac97cf265e3a59bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38957 } }
I20260928 22:56:01.522238 26635 raft_consensus.cc:388] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:56:01.522266 26635 raft_consensus.cc:482] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:56:01.522344 26635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:01.523039 26635 raft_consensus.cc:504] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b069d890279d4821ac97cf265e3a59bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38957 } }
I20260928 22:56:01.523151 26635 leader_election.cc:302] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b069d890279d4821ac97cf265e3a59bc; no voters:
I20260928 22:56:01.523342 26635 leader_election.cc:288] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:56:01.523452 26640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:56:01.523655 26635 sys_catalog.cc:564] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:56:01.523682 26640 raft_consensus.cc:686] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 1 LEADER]: Becoming Leader. State: Replica: b069d890279d4821ac97cf265e3a59bc, State: Running, Role: LEADER
I20260928 22:56:01.523964 26640 consensus_queue.cc:237] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b069d890279d4821ac97cf265e3a59bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38957 } }
I20260928 22:56:01.524400 26640 sys_catalog.cc:455] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b069d890279d4821ac97cf265e3a59bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b069d890279d4821ac97cf265e3a59bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38957 } } }
I20260928 22:56:01.524550 26640 sys_catalog.cc:458] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:01.524729 26640 sys_catalog.cc:455] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b069d890279d4821ac97cf265e3a59bc. Latest consensus state: current_term: 1 leader_uuid: "b069d890279d4821ac97cf265e3a59bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b069d890279d4821ac97cf265e3a59bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38957 } } }
I20260928 22:56:01.524840 26640 sys_catalog.cc:458] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:01.525007 26650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:56:01.525211 26635 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 22:56:01.525651 26650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:56:01 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:01.535482 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:01.518993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.14.14.190:38957)
0928 22:56:01.519165 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:01.519174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:01.519210 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:56:01.519768 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:01.519776 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:01.519848 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:01.519998 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:01.520820 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:01.522297 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:01.523448 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:01.524088 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:01.524206 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:01.527402 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:01.527406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:01.527407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:01.527419 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:01.532441 (+ 5022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:01.532981 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:01.532985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:01.532986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:01.533137 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:01.533424 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:01.533425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:01.533426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:01.533960 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:01.534574 (+ 614us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:01.534652 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 22:56:01.534850 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260928 22:56:01.540355 26650 catalog_manager.cc:1347] Generated new cluster ID: 7b7dcc65d564464d89d5aaf097e1848e
I20260928 22:56:01.540421 26650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:56:01.587400 26650 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:56:01.588336 26650 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:56:01.602478 26650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc: Generated new TSK 0
I20260928 22:56:01.602783 26650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:56:02.260725 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.14.148:58057/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38957
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:02.383188 26680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:02.383498 26680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:02.383545 26680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:02.385387 26680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:02.385465 26680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:02.385516 26680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:56:02.387928 26680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.14.148:58057/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:02.388221 26680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:02.388566 26680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:02.395431 26688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:02.395567 26685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:02.395457 26686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:02.396221 26680 server_base.cc:1034] running on GCE node
I20260928 22:56:02.396454 26680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:02.396761 26680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:02.397957 26680 hybrid_clock.cc:648] HybridClock initialized: now 1790636162397930 us; error 41 us; skew 500 ppm
Sep 28 22:56:02 dist-test-slave-kzfr krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636162, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:02.401203 26680 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:56:02.401796 26680 webserver.cc:460] Webserver started at http://127.14.14.129:45317/ using document root <none> and password file <none>
I20260928 22:56:02.402050 26680 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:02.402120 26680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:02.402262 26680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:56:02 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636162, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
I20260928 22:56:02.418239 26680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "072ef99cb67549efa45e527cd16d6479"
format_stamp: "Formatted at 2026-09-28 22:56:02 on dist-test-slave-kzfr"
server_key: "f1a644861718dd58143adff69b342eb1"
server_key_iv: "ee7de822e279435e15b21653159ff63b"
server_key_version: "kuduclusterkey@0"
I20260928 22:56:02.418809 26680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "072ef99cb67549efa45e527cd16d6479"
format_stamp: "Formatted at 2026-09-28 22:56:02 on dist-test-slave-kzfr"
server_key: "f1a644861718dd58143adff69b342eb1"
server_key_iv: "ee7de822e279435e15b21653159ff63b"
server_key_version: "kuduclusterkey@0"
I20260928 22:56:02.420755 26680 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 22:56:02.436782 26695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:02.436975 26680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:56:02.437088 26680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "072ef99cb67549efa45e527cd16d6479"
format_stamp: "Formatted at 2026-09-28 22:56:02 on dist-test-slave-kzfr"
server_key: "f1a644861718dd58143adff69b342eb1"
server_key_iv: "ee7de822e279435e15b21653159ff63b"
server_key_version: "kuduclusterkey@0"
I20260928 22:56:02.437201 26680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:02.460927 26680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:02.463644 26680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:02.463886 26680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:02.464217 26680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:02.464660 26680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:02.464833 26680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:02.465119 26680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:02.465310 26680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:02.472175 26680 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:34361
I20260928 22:56:02.472671 26680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:56:02.474013 26810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:34361 every 8 connection(s)
I20260928 22:56:02.474785 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 26680
I20260928 22:56:02.474885 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:56:02.475085 14394 external_mini_cluster.cc:1428] Setting key db8c6eac3d32f7723e10f5dcb11e049b
Sep 28 22:56:02 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636162, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
I20260928 22:56:02.494284 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:02.473847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:52645 (local address 127.14.14.190:38957)
0928 22:56:02.474225 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:02.474227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:02.474451 (+ 224us) server_negotiation.cc:407] Connection header received
0928 22:56:02.475188 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:02.475189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:02.475204 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:02.475275 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:02.484517 (+ 9242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:02.485305 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:02.487155 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:02.487463 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:02.487557 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:02.490407 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:02.490409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:02.490409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:02.490418 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:02.492275 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:02.492930 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:02.492931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:02.492931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:02.492997 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:02.493452 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:02.493453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:02.493454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:02.493634 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:02.493711 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:02.494083 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 22:56:02.494175 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":81,"threads_started":1}
I20260928 22:56:02.495155 26814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:02.473601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38957 (local address 127.14.14.129:52645)
0928 22:56:02.474287 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:02.474331 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:02.475012 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:02.475419 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:02.475424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:02.475670 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:02.484354 (+ 8684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:02.484365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:02.485434 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:02.485435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:02.487033 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:02.487037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:02.487648 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:02.487649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:02.487826 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:02.488499 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:02.488510 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:02.490264 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:02.492510 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:02.492516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:02.492519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:02.492832 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:02.493065 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:02.493066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:02.493066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:02.493347 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:02.493713 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:02.493720 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:02.493894 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 22:56:02.494012 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 22:56:02.494899 (+ 887us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":83,"threads_started":1}
I20260928 22:56:02.495995 26811 heartbeater.cc:347] Connected to a master server at 127.14.14.190:38957
I20260928 22:56:02.496107 26811 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:02.496323 26811 heartbeater.cc:511] Master 127.14.14.190:38957 requested a full tablet report, sending...
I20260928 22:56:02.496937 26599 ts_manager.cc:194] Registered new tserver with Master: 072ef99cb67549efa45e527cd16d6479 (127.14.14.129:34361)
I20260928 22:56:02.497830 26599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:52645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:56:02.517328 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.14.148:58057/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38957
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:02.635469 26817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:02.635849 26817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:02.635953 26817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:02.637838 26817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:02.638077 26817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:02.638202 26817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:56:02.640460 26817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.14.148:58057/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:02.641007 26817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:02.641340 26817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:02.648114 26827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:02.648944 26825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:02.649214 26817 server_base.cc:1034] running on GCE node
W20260928 22:56:02.649528 26824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:02.649863 26817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:02.650127 26817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:02.651298 26817 hybrid_clock.cc:648] HybridClock initialized: now 1790636162651279 us; error 33 us; skew 500 ppm
Sep 28 22:56:02 dist-test-slave-kzfr krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636162, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:02.654668 26817 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:56:02.655212 26817 webserver.cc:460] Webserver started at http://127.14.14.130:39209/ using document root <none> and password file <none>
I20260928 22:56:02.655454 26817 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:02.655500 26817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:02.655622 26817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:56:02 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636162, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
I20260928 22:56:02.672463 26817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "02b8571732934781b79e43ef8ede894f"
format_stamp: "Formatted at 2026-09-28 22:56:02 on dist-test-slave-kzfr"
server_key: "8ac1221aa0ae47e14d027aece90d5d9e"
server_key_iv: "1e3936a08c7476fba99f58ab32c2f776"
server_key_version: "kuduclusterkey@0"
I20260928 22:56:02.673038 26817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "02b8571732934781b79e43ef8ede894f"
format_stamp: "Formatted at 2026-09-28 22:56:02 on dist-test-slave-kzfr"
server_key: "8ac1221aa0ae47e14d027aece90d5d9e"
server_key_iv: "1e3936a08c7476fba99f58ab32c2f776"
server_key_version: "kuduclusterkey@0"
Sep 28 22:56:02 dist-test-slave-kzfr krb5kdc[26123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790636162, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:02.676743 26817 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20260928 22:56:02.700316 26834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:02.700592 26817 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260928 22:56:02.700697 26817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "02b8571732934781b79e43ef8ede894f"
format_stamp: "Formatted at 2026-09-28 22:56:02 on dist-test-slave-kzfr"
server_key: "8ac1221aa0ae47e14d027aece90d5d9e"
server_key_iv: "1e3936a08c7476fba99f58ab32c2f776"
server_key_version: "kuduclusterkey@0"
I20260928 22:56:02.700796 26817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:02.728116 26817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:02.730530 26817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:02.730727 26817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:02.730973 26817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:02.731348 26817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:02.731426 26817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:02.731496 26817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:02.731559 26817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:02.740028 26817 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:34565
I20260928 22:56:02.740592 26817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:56:02.741091 26947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:34565 every 8 connection(s)
I20260928 22:56:02.744123 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 26817
I20260928 22:56:02.744243 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:56:02.744449 14394 external_mini_cluster.cc:1428] Setting key a0eb08308a846dcb672850c6c32777b4
Sep 28 22:56:02 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636162, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:02.753841 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:02.742204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:33765 (local address 127.14.14.190:38957)
0928 22:56:02.742251 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:02.742253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:02.742597 (+ 344us) server_negotiation.cc:407] Connection header received
0928 22:56:02.743329 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:02.743331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:02.743347 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:02.743378 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:02.744263 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:02.745027 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:02.747124 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:02.747461 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:02.747505 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:02.750283 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:02.750285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:02.750285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:02.750294 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:02.752154 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:02.752647 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:02.752648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:02.752649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:02.752723 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:02.753017 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:02.753018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:02.753018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:02.753165 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:02.753210 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:02.753619 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 22:56:02.753713 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:56:02.754635 26950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:02.741930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38957 (local address 127.14.14.130:33765)
0928 22:56:02.742469 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:02.742512 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:02.743155 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:02.743561 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:02.743565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:02.743808 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:02.744087 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:02.744093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:02.745675 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:02.745685 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:02.747016 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:02.747019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:02.747606 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:02.747607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:02.747772 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:02.748390 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:02.748401 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:02.750168 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:02.752250 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:02.752256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:02.752259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:02.752557 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:02.752803 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:02.752804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:02.752804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:02.752934 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:02.753259 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:02.753265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:02.753433 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 22:56:02.754209 (+ 776us) client_negotiation.cc:240] Negotiation successful
0928 22:56:02.754378 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
I20260928 22:56:02.755313 26948 heartbeater.cc:347] Connected to a master server at 127.14.14.190:38957
I20260928 22:56:02.755403 26948 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:02.755589 26948 heartbeater.cc:511] Master 127.14.14.190:38957 requested a full tablet report, sending...
I20260928 22:56:02.756007 26599 ts_manager.cc:194] Registered new tserver with Master: 02b8571732934781b79e43ef8ede894f (127.14.14.130:34565)
I20260928 22:56:02.756469 26599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:33765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:56:02.780242 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.14.148:58057/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38957
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:02.896142 26954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:02.896422 26954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:02.896459 26954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:02.898250 26954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:02.898305 26954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:02.898343 26954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:56:02.900470 26954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.14.148:58057/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:02.900708 26954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:02.901016 26954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:02.907816 26960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:02.907830 26962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:02.908061 26954 server_base.cc:1034] running on GCE node
W20260928 22:56:02.908324 26959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:02.908595 26954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:02.908877 26954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:02.910054 26954 hybrid_clock.cc:648] HybridClock initialized: now 1790636162910032 us; error 34 us; skew 500 ppm
Sep 28 22:56:02 dist-test-slave-kzfr krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636162, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:02.913410 26954 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:56:02.913961 26954 webserver.cc:460] Webserver started at http://127.14.14.131:46869/ using document root <none> and password file <none>
I20260928 22:56:02.914273 26954 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:02.914345 26954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:02.914525 26954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:56:02 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636162, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
I20260928 22:56:02.931834 26954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "3c23f12f5bff4db2b542df487b6a1fc0"
format_stamp: "Formatted at 2026-09-28 22:56:02 on dist-test-slave-kzfr"
server_key: "5c1c36a1406084f4c02a0f4d9a3b4d44"
server_key_iv: "aedab374582d90241e40da95a0210df1"
server_key_version: "kuduclusterkey@0"
I20260928 22:56:02.932339 26954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "3c23f12f5bff4db2b542df487b6a1fc0"
format_stamp: "Formatted at 2026-09-28 22:56:02 on dist-test-slave-kzfr"
server_key: "5c1c36a1406084f4c02a0f4d9a3b4d44"
server_key_iv: "aedab374582d90241e40da95a0210df1"
server_key_version: "kuduclusterkey@0"
I20260928 22:56:02.937489 26954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260928 22:56:02.961723 26969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:02.962049 26954 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260928 22:56:02.962162 26954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "3c23f12f5bff4db2b542df487b6a1fc0"
format_stamp: "Formatted at 2026-09-28 22:56:02 on dist-test-slave-kzfr"
server_key: "5c1c36a1406084f4c02a0f4d9a3b4d44"
server_key_iv: "aedab374582d90241e40da95a0210df1"
server_key_version: "kuduclusterkey@0"
I20260928 22:56:02.962281 26954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:02.981813 26954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:02.984302 26954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:02.984444 26954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:02.984689 26954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:02.985081 26954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:02.985128 26954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:02.985158 26954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:02.985176 26954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:02.992357 26954 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:33083
I20260928 22:56:02.992540 27082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:33083 every 8 connection(s)
I20260928 22:56:02.992955 26954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:56:02.999091 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 26954
I20260928 22:56:02.999220 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:56:02.999421 14394 external_mini_cluster.cc:1428] Setting key 76361c8b6a4aaedeea002567b011676e
Sep 28 22:56:03 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636162, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:03.025137 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:03.006433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:46621 (local address 127.14.14.190:38957)
0928 22:56:03.006479 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:03.006481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:03.006696 (+ 215us) server_negotiation.cc:407] Connection header received
0928 22:56:03.008057 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:03.008058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:03.008070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:03.008101 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:03.008960 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.009753 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.011673 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.011965 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.012011 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:03.020714 (+ 8703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:03.020716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:03.020717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:03.020726 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:03.022558 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:03.023060 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:03.023062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:03.023062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:03.023139 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:03.024230 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:03.024232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:03.024232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:03.024390 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:03.024435 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:03.024936 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 22:56:03.025033 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:56:03.027305 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:03.006116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38957 (local address 127.14.14.131:46621)
0928 22:56:03.006562 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:03.006613 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:03.007926 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:03.008302 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:03.008307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:03.008574 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:03.008852 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:03.008858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.009865 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.009866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:03.011570 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:03.011573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.012117 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.012118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:03.012271 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:03.013331 (+ 1060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:03.013341 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:03.020576 (+ 7235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:03.022658 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:03.022666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:03.022669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:03.022970 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:03.023970 (+ 1000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:03.023971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:03.023971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:03.024135 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:03.024481 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:03.024487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:03.024745 (+ 258us) client_negotiation.cc:769] Sending connection context
0928 22:56:03.025316 (+ 571us) client_negotiation.cc:240] Negotiation successful
0928 22:56:03.027037 (+ 1721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":115,"threads_started":1}
I20260928 22:56:03.027875 27083 heartbeater.cc:347] Connected to a master server at 127.14.14.190:38957
I20260928 22:56:03.027969 27083 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:03.028213 27083 heartbeater.cc:511] Master 127.14.14.190:38957 requested a full tablet report, sending...
I20260928 22:56:03.028625 26599 ts_manager.cc:194] Registered new tserver with Master: 3c23f12f5bff4db2b542df487b6a1fc0 (127.14.14.131:33083)
I20260928 22:56:03.029114 26599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:46621
I20260928 22:56:03.034603 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:56:03.035315 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 26680
I20260928 22:56:03.041229 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 26817
I20260928 22:56:03.048669 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 26954
I20260928 22:56:03.056150 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 26567
I20260928 22:56:03.062505 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:38957
--webserver_interface=127.14.14.190
--webserver_port=38019
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:38957
--ranger_config_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.14.148:58057/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.14.148:58057/kms with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:03.143548 27086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:03.143757 27086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:03.143780 27086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:03.144851 27086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:56:03.144884 27086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:03.144899 27086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:03.144912 27086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:56:03.144924 27086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:56:03.146481 27086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.14.148:58057/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:38957
--ranger_config_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=38019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:03.146660 27086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:03.146847 27086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:03.151743 27092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:03.151786 27094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:03.151870 27086 server_base.cc:1034] running on GCE node
W20260928 22:56:03.151752 27091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:03.152161 27086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:03.152456 27086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:03.153651 27086 hybrid_clock.cc:648] HybridClock initialized: now 1790636163153632 us; error 37 us; skew 500 ppm
Sep 28 22:56:03 dist-test-slave-kzfr krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636163, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:03.156003 27086 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:56:03.156473 27086 webserver.cc:460] Webserver started at http://127.14.14.190:38019/ using document root <none> and password file <none>
I20260928 22:56:03.156684 27086 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:03.156759 27086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:56:03 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636163, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
I20260928 22:56:03.204398 27086 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 22:56:03.205346 27102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:03.205513 27086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:56:03.205653 27086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "b069d890279d4821ac97cf265e3a59bc"
format_stamp: "Formatted at 2026-09-28 22:56:01 on dist-test-slave-kzfr"
server_key: "9f4f486e299c99dcc797a36adb8fe7cb"
server_key_iv: "c39721f7c1b7bd5f14b951e90e31829a"
server_key_version: "kuduclusterkey@0"
I20260928 22:56:03.206034 27086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:03.264532 27086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:03.266592 27086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:03.266731 27086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:03.271446 27086 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:38957
I20260928 22:56:03.271529 27154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:38957 every 8 connection(s)
I20260928 22:56:03.271908 27086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:56:03.273129 27155 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:56:03.273865 27155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc: Bootstrap starting.
I20260928 22:56:03.278805 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 27086
I20260928 22:56:03.279130 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.14.148:58057/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:34361
--local_ip_for_outbound_sockets=127.14.14.129
--tserver_master_addrs=127.14.14.190:38957
--webserver_port=45317
--webserver_interface=127.14.14.129
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.14.148:58057/kms with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:56:03.282308 27155 log.cc:824] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:03.284672 27155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:56:03.285010 27155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc: Bootstrap complete.
I20260928 22:56:03.289175 27155 raft_consensus.cc:348] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b069d890279d4821ac97cf265e3a59bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38957 } }
I20260928 22:56:03.289568 27155 raft_consensus.cc:729] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b069d890279d4821ac97cf265e3a59bc, State: Initialized, Role: FOLLOWER
I20260928 22:56:03.289824 27155 consensus_queue.cc:260] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b069d890279d4821ac97cf265e3a59bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38957 } }
I20260928 22:56:03.289921 27155 raft_consensus.cc:388] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:56:03.290001 27155 raft_consensus.cc:482] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:56:03.290082 27155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:56:03.291534 27155 raft_consensus.cc:504] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b069d890279d4821ac97cf265e3a59bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38957 } }
I20260928 22:56:03.291638 27155 leader_election.cc:302] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b069d890279d4821ac97cf265e3a59bc; no voters:
I20260928 22:56:03.291862 27155 leader_election.cc:288] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 22:56:03.291925 27159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:56:03.292126 27159 raft_consensus.cc:686] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [term 2 LEADER]: Becoming Leader. State: Replica: b069d890279d4821ac97cf265e3a59bc, State: Running, Role: LEADER
I20260928 22:56:03.292163 27155 sys_catalog.cc:564] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:56:03.292567 27159 consensus_queue.cc:237] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b069d890279d4821ac97cf265e3a59bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38957 } }
I20260928 22:56:03.292894 27159 sys_catalog.cc:455] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b069d890279d4821ac97cf265e3a59bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b069d890279d4821ac97cf265e3a59bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38957 } } }
I20260928 22:56:03.293020 27159 sys_catalog.cc:458] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:03.293171 27159 sys_catalog.cc:455] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b069d890279d4821ac97cf265e3a59bc. Latest consensus state: current_term: 2 leader_uuid: "b069d890279d4821ac97cf265e3a59bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b069d890279d4821ac97cf265e3a59bc" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 38957 } } }
I20260928 22:56:03.293334 27159 sys_catalog.cc:458] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:03.293470 27168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:56:03.293779 27155 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 22:56:03.294018 27168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:56:03.294126 27168 catalog_manager.cc:1259] Loaded cluster ID: 7b7dcc65d564464d89d5aaf097e1848e
I20260928 22:56:03.294200 27168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:56:03.298120 27168 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:56:03.299558 27168 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc: Loaded TSK: 0
I20260928 22:56:03.299938 27168 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:56:03.396626 27157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:03.396996 27157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:03.397092 27157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:03.398880 27157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:03.399091 27157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:03.399187 27157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:56:03.401366 27157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.14.148:58057/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:34361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=45317
--tserver_master_addrs=127.14.14.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:03.401881 27157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:03.402212 27157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:03.408667 27193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:03.409071 27192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:03.409109 27195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:03.410801 27157 server_base.cc:1034] running on GCE node
I20260928 22:56:03.411115 27157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:03.411422 27157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:03.416098 27157 hybrid_clock.cc:648] HybridClock initialized: now 1790636163416060 us; error 50 us; skew 500 ppm
Sep 28 22:56:03 dist-test-slave-kzfr krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636163, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:03.419510 27157 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:56:03.420099 27157 webserver.cc:460] Webserver started at http://127.14.14.129:45317/ using document root <none> and password file <none>
I20260928 22:56:03.420434 27157 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:03.420573 27157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:56:03 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636163, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
I20260928 22:56:03.441968 27157 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 22:56:03.443073 27203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:03.443280 27157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:56:03.443362 27157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "072ef99cb67549efa45e527cd16d6479"
format_stamp: "Formatted at 2026-09-28 22:56:02 on dist-test-slave-kzfr"
server_key: "f1a644861718dd58143adff69b342eb1"
server_key_iv: "ee7de822e279435e15b21653159ff63b"
server_key_version: "kuduclusterkey@0"
I20260928 22:56:03.443841 27157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:03.474406 27157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:03.476812 27157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:03.477027 27157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:03.477270 27157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:03.477717 27157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:03.477821 27157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:03.477921 27157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:03.477991 27157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:03.484360 27157 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:34361
I20260928 22:56:03.484895 27157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:56:03.489360 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 27157
I20260928 22:56:03.489718 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.14.148:58057/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:34565
--local_ip_for_outbound_sockets=127.14.14.130
--tserver_master_addrs=127.14.14.190:38957
--webserver_port=39209
--webserver_interface=127.14.14.130
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.14.148:58057/kms with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:56:03.498979 27316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:34361 every 8 connection(s)
Sep 28 22:56:03 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636163, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:03.513083 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:03.499936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38957 (local address 127.14.14.129:51151)
0928 22:56:03.500291 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:03.500349 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:03.501030 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:03.501872 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:03.501878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:03.502114 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:03.502404 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:03.502410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.504075 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.504076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:03.505361 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:03.505364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.506349 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.506351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:03.506473 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:03.507108 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:03.507119 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:03.508868 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:03.511148 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:03.511154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:03.511157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:03.511451 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:03.511738 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:03.511738 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:03.511739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:03.511882 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:03.512399 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:03.512405 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:03.512557 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 22:56:03.512672 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 22:56:03.512843 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":98,"threads_started":1}
I20260928 22:56:03.514211 27321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:03.500015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:51151 (local address 127.14.14.190:38957)
0928 22:56:03.501271 (+ 1256us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:03.501281 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:03.501322 (+ 41us) server_negotiation.cc:407] Connection header received
0928 22:56:03.501473 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:03.501479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:03.501538 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:03.501705 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:03.502541 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.503951 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.505474 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.506126 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.506232 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:03.508993 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:03.508998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:03.508998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:03.509011 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:03.511052 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:03.511545 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:03.511549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:03.511549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:03.511660 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:03.511962 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:03.511963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:03.511963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:03.512273 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:03.512358 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:03.513857 (+ 1499us) server_negotiation.cc:299] Negotiation successful
0928 22:56:03.514006 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1184,"thread_start_us":98,"threads_started":1}
I20260928 22:56:03.515170 27317 heartbeater.cc:347] Connected to a master server at 127.14.14.190:38957
I20260928 22:56:03.515269 27317 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:03.515492 27317 heartbeater.cc:511] Master 127.14.14.190:38957 requested a full tablet report, sending...
W20260928 22:56:03.516386 27317 heartbeater.cc:644] Failed to heartbeat to 127.14.14.190:38957 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 22:56:03.516453 27317 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:03.516512 27317 heartbeater.cc:511] Master 127.14.14.190:38957 requested a full tablet report, sending...
I20260928 22:56:03.516927 27317 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:03.516999 27317 heartbeater.cc:511] Master 127.14.14.190:38957 requested a full tablet report, sending...
W20260928 22:56:03.517329 27317 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:56:03.611825 27319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:03.612187 27319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:03.612282 27319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:03.614044 27319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:03.614260 27319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:03.614356 27319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:56:03.616496 27319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.14.148:58057/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:34565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=39209
--tserver_master_addrs=127.14.14.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:03.616755 27319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:03.617051 27319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:03.623837 27327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:03.624202 27326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:03.624429 27329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:03.626621 27319 server_base.cc:1034] running on GCE node
I20260928 22:56:03.627003 27319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:03.627313 27319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:03.628468 27319 hybrid_clock.cc:648] HybridClock initialized: now 1790636163628438 us; error 50 us; skew 500 ppm
Sep 28 22:56:03 dist-test-slave-kzfr krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636163, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:03.631623 27319 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:56:03.632138 27319 webserver.cc:460] Webserver started at http://127.14.14.130:39209/ using document root <none> and password file <none>
I20260928 22:56:03.632426 27319 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:03.632509 27319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:56:03 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636163, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
I20260928 22:56:03.649276 27319 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 22:56:03.650072 27337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:03.650239 27319 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:03.650336 27319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "02b8571732934781b79e43ef8ede894f"
format_stamp: "Formatted at 2026-09-28 22:56:02 on dist-test-slave-kzfr"
server_key: "8ac1221aa0ae47e14d027aece90d5d9e"
server_key_iv: "1e3936a08c7476fba99f58ab32c2f776"
server_key_version: "kuduclusterkey@0"
I20260928 22:56:03.651444 27319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:03.661111 27319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:03.663511 27319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:03.663673 27319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:03.663890 27319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:03.664217 27319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:03.664278 27319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:03.664320 27319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:03.664346 27319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:03.670225 27319 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:34565
I20260928 22:56:03.670612 27319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:56:03.674086 27450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:34565 every 8 connection(s)
I20260928 22:56:03.675928 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 27319
I20260928 22:56:03.676205 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.14.148:58057/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:33083
--local_ip_for_outbound_sockets=127.14.14.131
--tserver_master_addrs=127.14.14.190:38957
--webserver_port=46869
--webserver_interface=127.14.14.131
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.14.148:58057/kms with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:56:03 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636163, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:03.687090 27321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:03.674743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:58445 (local address 127.14.14.190:38957)
0928 22:56:03.674803 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:03.674805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:03.675033 (+ 228us) server_negotiation.cc:407] Connection header received
0928 22:56:03.675742 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:03.675743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:03.675755 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:03.675786 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:03.676634 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.677405 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.678941 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.679266 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.679345 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:03.683416 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:03.683418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:03.683418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:03.683426 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:03.685265 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:03.685831 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:03.685833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:03.685833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:03.685898 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:03.686216 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:03.686217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:03.686218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:03.686382 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:03.686442 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:03.686852 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 22:56:03.686948 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:56:03.687711 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:03.674728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38957 (local address 127.14.14.130:58445)
0928 22:56:03.674891 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:03.674941 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:03.675605 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:03.675977 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:03.675981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:03.676212 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:03.676509 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:03.676516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.677527 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.677528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:03.678827 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:03.678830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.679430 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.679432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:03.679570 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:03.681398 (+ 1828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:03.681408 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:03.683290 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:03.685365 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:03.685372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:03.685374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:03.685732 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:03.685982 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:03.685983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:03.685983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:03.686131 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:03.686482 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:03.686488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:03.686678 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 22:56:03.687258 (+ 580us) client_negotiation.cc:240] Negotiation successful
0928 22:56:03.687420 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260928 22:56:03.688241 27451 heartbeater.cc:347] Connected to a master server at 127.14.14.190:38957
I20260928 22:56:03.688362 27451 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:03.688570 27451 heartbeater.cc:511] Master 127.14.14.190:38957 requested a full tablet report, sending...
W20260928 22:56:03.689267 27451 heartbeater.cc:644] Failed to heartbeat to 127.14.14.190:38957 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 22:56:03.689339 27451 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:03.689407 27451 heartbeater.cc:511] Master 127.14.14.190:38957 requested a full tablet report, sending...
I20260928 22:56:03.689848 27451 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:03.689934 27451 heartbeater.cc:511] Master 127.14.14.190:38957 requested a full tablet report, sending...
W20260928 22:56:03.690264 27451 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:56:03.780725 27454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:03.780998 27454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:03.781036 27454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:03.782963 27454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:03.783023 27454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:03.783069 27454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:56:03.784639 27454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.14.148:58057/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:33083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=46869
--tserver_master_addrs=127.14.14.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:03.785065 27454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:03.785367 27454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:03.792008 27460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:03.792013 27459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:03.792084 27462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:03.793447 27454 server_base.cc:1034] running on GCE node
I20260928 22:56:03.793757 27454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:03.794077 27454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:03.795418 27454 hybrid_clock.cc:648] HybridClock initialized: now 1790636163795399 us; error 31 us; skew 500 ppm
Sep 28 22:56:03 dist-test-slave-kzfr krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636163, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:03.798789 27454 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:56:03.799414 27454 webserver.cc:460] Webserver started at http://127.14.14.131:46869/ using document root <none> and password file <none>
I20260928 22:56:03.799671 27454 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:03.799763 27454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:56:03 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636163, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
I20260928 22:56:03.821274 27454 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 22:56:03.822167 27470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:03.822386 27454 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:56:03.822479 27454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "3c23f12f5bff4db2b542df487b6a1fc0"
format_stamp: "Formatted at 2026-09-28 22:56:02 on dist-test-slave-kzfr"
server_key: "5c1c36a1406084f4c02a0f4d9a3b4d44"
server_key_iv: "aedab374582d90241e40da95a0210df1"
server_key_version: "kuduclusterkey@0"
I20260928 22:56:03.823685 27454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:03.839501 27454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:03.842101 27454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:03.842314 27454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:03.842628 27454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:03.843071 27454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:03.843134 27454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:03.843180 27454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:03.843204 27454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:03.851179 27454 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:33083
I20260928 22:56:03.851752 27454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:56:03.854542 27583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:33083 every 8 connection(s)
I20260928 22:56:03.855404 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 27454
Sep 28 22:56:03 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636163, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:03.869674 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:03.855857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.14.14.190:38957)
0928 22:56:03.856003 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:03.856005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:03.859373 (+ 3368us) server_negotiation.cc:407] Connection header received
0928 22:56:03.859394 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:03.859394 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:03.859407 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:03.859468 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:03.861317 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.862068 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.862999 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.863311 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.863377 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:03.865355 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:03.865357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:03.865357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:03.865365 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:03.867599 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:03.868315 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:03.868316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:03.868317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:03.868379 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:03.869057 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:03.869058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:03.869058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:03.869192 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:03.869374 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:03.869441 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 22:56:03.869526 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260928 22:56:03.870702 27321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:03.855232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:40593 (local address 127.14.14.190:38957)
0928 22:56:03.855294 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:03.855296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:03.857969 (+ 2673us) server_negotiation.cc:407] Connection header received
0928 22:56:03.858620 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:03.858621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:03.858634 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:03.858685 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:03.859358 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.859899 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.861294 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.861570 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.861728 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:03.866099 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:03.866100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:03.866101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:03.866108 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:03.867979 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:03.868705 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:03.868706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:03.868706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:03.868764 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:03.869036 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:03.869037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:03.869037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:03.869198 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:03.869261 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:03.870528 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0928 22:56:03.870605 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:56:03.871079 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:03.855231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:38957 (local address 127.14.14.131:40593)
0928 22:56:03.857824 (+ 2593us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:03.857863 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:03.858492 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:03.858828 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:03.858834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:03.859040 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:03.859254 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:03.859258 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.859990 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.859991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:03.861194 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:03.861196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:03.861820 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:03.861821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:03.861964 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:03.863093 (+ 1129us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:03.863102 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:03.865985 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:03.868335 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:03.868342 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:03.868345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:03.868615 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:03.868836 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:03.868837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:03.868838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:03.868959 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:03.869699 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:03.869706 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:03.870299 (+ 593us) client_negotiation.cc:769] Sending connection context
0928 22:56:03.870509 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 22:56:03.870670 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2553,"thread_start_us":110,"threads_started":1}
I20260928 22:56:03.871549 27584 heartbeater.cc:347] Connected to a master server at 127.14.14.190:38957
I20260928 22:56:03.871680 27584 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:03.871878 27584 heartbeater.cc:511] Master 127.14.14.190:38957 requested a full tablet report, sending...
W20260928 22:56:03.872534 27584 heartbeater.cc:644] Failed to heartbeat to 127.14.14.190:38957 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 22:56:03.872617 27584 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:03.872689 27584 heartbeater.cc:511] Master 127.14.14.190:38957 requested a full tablet report, sending...
I20260928 22:56:03.872960 27584 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:03.873032 27584 heartbeater.cc:511] Master 127.14.14.190:38957 requested a full tablet report, sending...
W20260928 22:56:03.873309 27584 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 22:56:04 dist-test-slave-kzfr krb5kdc[26123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790636164, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:04.517517 27317 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:04.517674 27317 heartbeater.cc:511] Master 127.14.14.190:38957 requested a full tablet report, sending...
I20260928 22:56:04.518088 27119 ts_manager.cc:194] Registered new tserver with Master: 072ef99cb67549efa45e527cd16d6479 (127.14.14.129:34361)
I20260928 22:56:04.519085 27119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:51151
I20260928 22:56:04.690454 27451 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:04.690632 27451 heartbeater.cc:511] Master 127.14.14.190:38957 requested a full tablet report, sending...
I20260928 22:56:04.690940 27119 ts_manager.cc:194] Registered new tserver with Master: 02b8571732934781b79e43ef8ede894f (127.14.14.130:34565)
I20260928 22:56:04.691355 27119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:58445
Sep 28 22:56:04 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636164, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
I20260928 22:56:04.873514 27584 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:04.873701 27584 heartbeater.cc:511] Master 127.14.14.190:38957 requested a full tablet report, sending...
I20260928 22:56:04.874532 27119 ts_manager.cc:194] Registered new tserver with Master: 3c23f12f5bff4db2b542df487b6a1fc0 (127.14.14.131:33083)
I20260928 22:56:04.875082 27119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:40593
I20260928 22:56:04.882678 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:56:04.893751 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:04.883963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.14.14.190:38957)
0928 22:56:04.884331 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:04.884333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:04.884796 (+ 463us) server_negotiation.cc:407] Connection header received
0928 22:56:04.884822 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:04.884823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:04.884838 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:04.884896 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:04.885695 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:04.886480 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:04.887480 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:04.887764 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:04.887831 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:04.889399 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:04.889401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:04.889401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:04.889408 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:04.892168 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:04.892636 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:04.892637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:04.892638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:04.892704 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:04.892927 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:04.892960 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:04.892960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:04.893093 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:04.893188 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:04.893523 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 22:56:04.893650 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":61,"threads_started":1}
I20260928 22:56:04.896526 27119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:56:04.898969 27119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:56:04.909569 27617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:04.902943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:34361 (local address 127.0.0.1:57968)
0928 22:56:04.904241 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:04.904281 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:04.904346 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:04.904813 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:04.904816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:04.904818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:04.904895 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:04.904896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:04.906452 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:04.906453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:04.907754 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:04.907757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:04.909205 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:04.909206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:04.909329 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:04.909343 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:56:04.909436 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 22:56:04.909486 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1269,"thread_start_us":78,"threads_started":1}
I20260928 22:56:04.910177 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:04.903204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.14.14.129:34361)
0928 22:56:04.904592 (+ 1388us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:04.904597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:04.904606 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:56:04.904642 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:04.904644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:04.904658 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:04.904725 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:04.904974 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:04.906355 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:04.907895 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:04.909036 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:04.909106 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:04.909162 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:04.910057 (+ 895us) server_negotiation.cc:299] Negotiation successful
0928 22:56:04.910099 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1352,"thread_start_us":62,"threads_started":1}
I20260928 22:56:04.911012 27249 tablet_service.cc:1467] Processing CreateTablet for tablet 82f00363bca34f99b49921f701c575fa (DEFAULT_TABLE table=test-table [id=4bfade49cb114b6e899149ad2ca6ddb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:56:04.911389 27249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f00363bca34f99b49921f701c575fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:04.911957 27615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:04.903593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:34565 (local address 127.0.0.1:50858)
0928 22:56:04.907953 (+ 4360us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:04.907965 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:04.908043 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:04.908314 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:04.908315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:04.908317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:04.908374 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:04.908384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:04.909600 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:04.909601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:04.910728 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:04.910731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:04.911735 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:04.911736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:04.911805 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:04.911809 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:56:04.911853 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:56:04.911888 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4313,"thread_start_us":55,"threads_started":1}
I20260928 22:56:04.912209 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:04.903790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.14.14.130:34565)
0928 22:56:04.904174 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:04.904179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:04.908105 (+ 3926us) server_negotiation.cc:407] Connection header received
0928 22:56:04.908146 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:04.908148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:04.908164 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:04.908234 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:04.908456 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:04.909486 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:04.910822 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:04.911579 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:04.911644 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:04.911704 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:04.912084 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 22:56:04.912121 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":63,"threads_started":1}
I20260928 22:56:04.912822 27385 tablet_service.cc:1467] Processing CreateTablet for tablet 82f00363bca34f99b49921f701c575fa (DEFAULT_TABLE table=test-table [id=4bfade49cb114b6e899149ad2ca6ddb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:56:04.913147 27385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f00363bca34f99b49921f701c575fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:04.913354 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:04.903433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:33083 (local address 127.0.0.1:55754)
0928 22:56:04.904444 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:04.904453 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:04.904489 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:04.909895 (+ 5406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:04.909896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:04.909898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:04.909929 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:04.909930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:04.911103 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:04.911104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:04.912138 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:04.912140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:04.913118 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:04.913119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:04.913197 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:04.913200 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:56:04.913255 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:56:04.913292 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"thread_start_us":78,"threads_started":1}
I20260928 22:56:04.913646 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:04.903533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.14.14.131:33083)
0928 22:56:04.909664 (+ 6131us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:04.909669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:04.909679 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:56:04.909713 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:04.909715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:04.909730 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:04.909811 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:04.910003 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:04.911013 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:04.912246 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:04.912952 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:04.913033 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:04.913082 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:04.913508 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 22:56:04.913552 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6095,"thread_start_us":67,"threads_started":1}
I20260928 22:56:04.916239 27619 tablet_bootstrap.cc:492] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479: Bootstrap starting.
I20260928 22:56:04.916848 27518 tablet_service.cc:1467] Processing CreateTablet for tablet 82f00363bca34f99b49921f701c575fa (DEFAULT_TABLE table=test-table [id=4bfade49cb114b6e899149ad2ca6ddb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:56:04.916949 27619 tablet_bootstrap.cc:654] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:04.917140 27518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f00363bca34f99b49921f701c575fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:04.917244 27619 log.cc:824] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:04.918018 27619 tablet_bootstrap.cc:492] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479: No bootstrap required, opened a new log
I20260928 22:56:04.918083 27619 ts_tablet_manager.cc:1397] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:56:04.920704 27622 tablet_bootstrap.cc:492] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0: Bootstrap starting.
I20260928 22:56:04.920639 27619 raft_consensus.cc:348] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072ef99cb67549efa45e527cd16d6479" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 34361 } } peers { permanent_uuid: "3c23f12f5bff4db2b542df487b6a1fc0" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33083 } } peers { permanent_uuid: "02b8571732934781b79e43ef8ede894f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34565 } }
I20260928 22:56:04.921270 27619 raft_consensus.cc:374] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:04.921393 27622 tablet_bootstrap.cc:654] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:04.921732 27622 log.cc:824] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:04.922022 27620 tablet_bootstrap.cc:492] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f: Bootstrap starting.
I20260928 22:56:04.922647 27620 tablet_bootstrap.cc:654] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:04.922762 27619 raft_consensus.cc:729] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 072ef99cb67549efa45e527cd16d6479, State: Initialized, Role: FOLLOWER
I20260928 22:56:04.922945 27620 log.cc:824] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:04.923044 27619 consensus_queue.cc:260] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072ef99cb67549efa45e527cd16d6479" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 34361 } } peers { permanent_uuid: "3c23f12f5bff4db2b542df487b6a1fc0" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33083 } } peers { permanent_uuid: "02b8571732934781b79e43ef8ede894f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34565 } }
I20260928 22:56:04.923393 27619 ts_tablet_manager.cc:1428] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 22:56:04.923547 27317 heartbeater.cc:503] Master 127.14.14.190:38957 was elected leader, sending a full tablet report...
I20260928 22:56:04.923667 27622 tablet_bootstrap.cc:492] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0: No bootstrap required, opened a new log
I20260928 22:56:04.923734 27622 ts_tablet_manager.cc:1397] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 22:56:04.926129 27622 raft_consensus.cc:348] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072ef99cb67549efa45e527cd16d6479" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 34361 } } peers { permanent_uuid: "3c23f12f5bff4db2b542df487b6a1fc0" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33083 } } peers { permanent_uuid: "02b8571732934781b79e43ef8ede894f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34565 } }
I20260928 22:56:04.926286 27622 raft_consensus.cc:374] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:04.926317 27622 raft_consensus.cc:729] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c23f12f5bff4db2b542df487b6a1fc0, State: Initialized, Role: FOLLOWER
I20260928 22:56:04.926424 27622 consensus_queue.cc:260] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072ef99cb67549efa45e527cd16d6479" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 34361 } } peers { permanent_uuid: "3c23f12f5bff4db2b542df487b6a1fc0" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33083 } } peers { permanent_uuid: "02b8571732934781b79e43ef8ede894f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34565 } }
I20260928 22:56:04.926648 27622 ts_tablet_manager.cc:1428] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 22:56:04.926685 27584 heartbeater.cc:503] Master 127.14.14.190:38957 was elected leader, sending a full tablet report...
I20260928 22:56:04.927253 27620 tablet_bootstrap.cc:492] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f: No bootstrap required, opened a new log
I20260928 22:56:04.927318 27620 ts_tablet_manager.cc:1397] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 22:56:04.929782 27620 raft_consensus.cc:348] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072ef99cb67549efa45e527cd16d6479" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 34361 } } peers { permanent_uuid: "3c23f12f5bff4db2b542df487b6a1fc0" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33083 } } peers { permanent_uuid: "02b8571732934781b79e43ef8ede894f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34565 } }
I20260928 22:56:04.929957 27620 raft_consensus.cc:374] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:04.929986 27620 raft_consensus.cc:729] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b8571732934781b79e43ef8ede894f, State: Initialized, Role: FOLLOWER
I20260928 22:56:04.930109 27620 consensus_queue.cc:260] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072ef99cb67549efa45e527cd16d6479" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 34361 } } peers { permanent_uuid: "3c23f12f5bff4db2b542df487b6a1fc0" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33083 } } peers { permanent_uuid: "02b8571732934781b79e43ef8ede894f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34565 } }
I20260928 22:56:04.930343 27620 ts_tablet_manager.cc:1428] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:56:04.930471 27451 heartbeater.cc:503] Master 127.14.14.190:38957 was elected leader, sending a full tablet report...
W20260928 22:56:05.000423 27318 tablet.cc:2369] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:56:05.108659 27585 tablet.cc:2369] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 22:56:05 dist-test-slave-kzfr krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636164, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for HTTP/127.14.14.148@KRBTEST.COM
W20260928 22:56:05.175890 27452 tablet.cc:2369] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:56:05.310145 27625 raft_consensus.cc:482] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:56:05.310317 27625 raft_consensus.cc:504] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072ef99cb67549efa45e527cd16d6479" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 34361 } } peers { permanent_uuid: "3c23f12f5bff4db2b542df487b6a1fc0" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33083 } } peers { permanent_uuid: "02b8571732934781b79e43ef8ede894f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34565 } }
I20260928 22:56:05.310735 27625 leader_election.cc:288] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c23f12f5bff4db2b542df487b6a1fc0 (127.14.14.131:33083), 02b8571732934781b79e43ef8ede894f (127.14.14.130:34565)
I20260928 22:56:05.315032 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:05.311391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:33501 (local address 127.14.14.130:34565)
0928 22:56:05.311436 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:05.311438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:05.311448 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:56:05.311479 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:05.311479 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:05.311489 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:05.311556 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:05.311769 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:05.312488 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:05.313987 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:05.314593 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:05.314648 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:05.314659 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:05.314912 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 22:56:05.314956 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:56:05.315505 27631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:05.311183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:34565 (local address 127.14.14.129:33501)
0928 22:56:05.311347 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:05.311356 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:05.311408 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:05.311583 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:05.311584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:05.311586 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:05.311674 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:05.311677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:05.312602 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:05.312603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:05.313873 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:05.313876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:05.314742 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:05.314743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:05.314824 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:05.314827 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:56:05.315346 (+ 519us) client_negotiation.cc:240] Negotiation successful
0928 22:56:05.315392 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260928 22:56:05.316027 27405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f00363bca34f99b49921f701c575fa" candidate_uuid: "072ef99cb67549efa45e527cd16d6479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b8571732934781b79e43ef8ede894f" is_pre_election: true
I20260928 22:56:05.316202 27405 raft_consensus.cc:2454] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 072ef99cb67549efa45e527cd16d6479 in term 0.
I20260928 22:56:05.316540 27205 leader_election.cc:302] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b8571732934781b79e43ef8ede894f, 072ef99cb67549efa45e527cd16d6479; no voters:
I20260928 22:56:05.316669 27625 raft_consensus.cc:2781] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:56:05.316708 27625 raft_consensus.cc:482] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:56:05.316730 27625 raft_consensus.cc:3037] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:05.317523 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:05.311052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:41593 (local address 127.14.14.131:33083)
0928 22:56:05.311103 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:05.311106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:05.311279 (+ 173us) server_negotiation.cc:407] Connection header received
0928 22:56:05.313773 (+ 2494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:05.313775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:05.313785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:05.313816 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:05.314031 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:05.314791 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:05.316173 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:05.316765 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:05.316820 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:05.316827 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:05.317412 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 22:56:05.317456 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:56:05.319137 27625 raft_consensus.cc:504] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072ef99cb67549efa45e527cd16d6479" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 34361 } } peers { permanent_uuid: "3c23f12f5bff4db2b542df487b6a1fc0" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33083 } } peers { permanent_uuid: "02b8571732934781b79e43ef8ede894f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34565 } }
I20260928 22:56:05.319273 27625 leader_election.cc:288] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [CANDIDATE]: Term 1 election: Requested vote from peers 3c23f12f5bff4db2b542df487b6a1fc0 (127.14.14.131:33083), 02b8571732934781b79e43ef8ede894f (127.14.14.130:34565)
I20260928 22:56:05.317762 27630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:05.310946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:33083 (local address 127.14.14.129:41593)
0928 22:56:05.311160 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:05.311173 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:05.311276 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:05.313884 (+ 2608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:05.313885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:05.313887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:05.313953 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:05.313955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:05.314901 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:05.314901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:05.316055 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:05.316058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:05.316888 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:05.316889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:05.316962 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:05.316966 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:56:05.317629 (+ 663us) client_negotiation.cc:240] Negotiation successful
0928 22:56:05.317674 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":108,"threads_started":1}
I20260928 22:56:05.319691 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f00363bca34f99b49921f701c575fa" candidate_uuid: "072ef99cb67549efa45e527cd16d6479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c23f12f5bff4db2b542df487b6a1fc0"
I20260928 22:56:05.319702 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f00363bca34f99b49921f701c575fa" candidate_uuid: "072ef99cb67549efa45e527cd16d6479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c23f12f5bff4db2b542df487b6a1fc0" is_pre_election: true
I20260928 22:56:05.319849 27521 raft_consensus.cc:3037] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:05.320775 27521 raft_consensus.cc:2454] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 072ef99cb67549efa45e527cd16d6479 in term 1.
I20260928 22:56:05.321033 27405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f00363bca34f99b49921f701c575fa" candidate_uuid: "072ef99cb67549efa45e527cd16d6479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b8571732934781b79e43ef8ede894f"
I20260928 22:56:05.321132 27405 raft_consensus.cc:3037] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:05.321317 27205 leader_election.cc:302] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 072ef99cb67549efa45e527cd16d6479, 3c23f12f5bff4db2b542df487b6a1fc0; no voters:
I20260928 22:56:05.321448 27625 raft_consensus.cc:2781] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:56:05.321625 27625 raft_consensus.cc:686] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [term 1 LEADER]: Becoming Leader. State: Replica: 072ef99cb67549efa45e527cd16d6479, State: Running, Role: LEADER
I20260928 22:56:05.321718 27625 consensus_queue.cc:237] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072ef99cb67549efa45e527cd16d6479" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 34361 } } peers { permanent_uuid: "3c23f12f5bff4db2b542df487b6a1fc0" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33083 } } peers { permanent_uuid: "02b8571732934781b79e43ef8ede894f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34565 } }
I20260928 22:56:05.322050 27405 raft_consensus.cc:2454] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 072ef99cb67549efa45e527cd16d6479 in term 1.
I20260928 22:56:05.322733 27119 catalog_manager.cc:5495] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 reported cstate change: term changed from 0 to 1, leader changed from <none> to 072ef99cb67549efa45e527cd16d6479 (127.14.14.129). New cstate: current_term: 1 leader_uuid: "072ef99cb67549efa45e527cd16d6479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072ef99cb67549efa45e527cd16d6479" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 34361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c23f12f5bff4db2b542df487b6a1fc0" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02b8571732934781b79e43ef8ede894f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260928 22:56:05.353479 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:05.350706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.14.14.129:34361)
0928 22:56:05.350755 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:05.350757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:05.350765 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:56:05.350849 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:05.350849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:05.350867 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:05.350931 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:56:05.351072 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:05.351551 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:05.352709 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:05.352921 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:05.352978 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:05.353165 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:56:05.353265 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:56:05.353337 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:05.353400 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 22:56:05.353421 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:56:05.355412 27521 raft_consensus.cc:1261] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0 [term 1 FOLLOWER]: Refusing update from remote peer 072ef99cb67549efa45e527cd16d6479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:56:05.355448 27405 raft_consensus.cc:1261] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f [term 1 FOLLOWER]: Refusing update from remote peer 072ef99cb67549efa45e527cd16d6479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:56:05.355746 27625 consensus_queue.cc:1035] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c23f12f5bff4db2b542df487b6a1fc0" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:56:05.355904 27632 consensus_queue.cc:1035] T 82f00363bca34f99b49921f701c575fa P 072ef99cb67549efa45e527cd16d6479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b8571732934781b79e43ef8ede894f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:56:05.358479 27642 mvcc.cc:205] Tried to move back new op lower bound from 7334445733292707840 to 7334445733158051840. Current Snapshot: MvccSnapshot[applied={T|T < 7334445733292707840}]
I20260928 22:56:05.359192 27644 mvcc.cc:205] Tried to move back new op lower bound from 7334445733292707840 to 7334445733158051840. Current Snapshot: MvccSnapshot[applied={T|T < 7334445733292707840}]
I20260928 22:56:05.360080 27643 mvcc.cc:205] Tried to move back new op lower bound from 7334445733292707840 to 7334445733158051840. Current Snapshot: MvccSnapshot[applied={T|T < 7334445733292707840 or (T in {7334445733292707840})}]
I20260928 22:56:05.361639 27118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:56:05.361739 27118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:56:05.362545 27118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b069d890279d4821ac97cf265e3a59bc: Sending DeleteTablet for 3 replicas of tablet 82f00363bca34f99b49921f701c575fa
I20260928 22:56:05.362861 27249 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f00363bca34f99b49921f701c575fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:05 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:57968
I20260928 22:56:05.362982 27518 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f00363bca34f99b49921f701c575fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:05 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:55754
I20260928 22:56:05.363209 27652 tablet_replica.cc:330] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0: stopping tablet replica
I20260928 22:56:05.363217 27385 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f00363bca34f99b49921f701c575fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:05 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:50858
I20260928 22:56:05.363320 27652 raft_consensus.cc:2229] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:56:05.363436 27652 raft_consensus.cc:2258] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:56:05.363457 27653 tablet_replica.cc:330] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f: stopping tablet replica
I20260928 22:56:05.363601 27653 raft_consensus.cc:2229] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:56:05.363680 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 27157
I20260928 22:56:05.363710 27653 raft_consensus.cc:2258] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:56:05.364900 27653 ts_tablet_manager.cc:1905] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:56:05.366503 27652 ts_tablet_manager.cc:1905] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:56:05.366716 27653 ts_tablet_manager.cc:1918] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:56:05.366791 27653 log.cc:1196] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/82f00363bca34f99b49921f701c575fa
I20260928 22:56:05.367063 27653 ts_tablet_manager.cc:1939] T 82f00363bca34f99b49921f701c575fa P 02b8571732934781b79e43ef8ede894f: Deleting consensus metadata
I20260928 22:56:05.367473 27104 catalog_manager.cc:4841] TS 02b8571732934781b79e43ef8ede894f (127.14.14.130:34565): tablet 82f00363bca34f99b49921f701c575fa (table test-table [id=4bfade49cb114b6e899149ad2ca6ddb0]) successfully deleted
I20260928 22:56:05.368093 27652 ts_tablet_manager.cc:1918] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:56:05.368168 27652 log.cc:1196] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/82f00363bca34f99b49921f701c575fa
I20260928 22:56:05.368444 27652 ts_tablet_manager.cc:1939] T 82f00363bca34f99b49921f701c575fa P 3c23f12f5bff4db2b542df487b6a1fc0: Deleting consensus metadata
I20260928 22:56:05.368884 27104 catalog_manager.cc:4841] TS 3c23f12f5bff4db2b542df487b6a1fc0 (127.14.14.131:33083): tablet 82f00363bca34f99b49921f701c575fa (table test-table [id=4bfade49cb114b6e899149ad2ca6ddb0]) successfully deleted
W20260928 22:56:05.371491 27103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:34361 (error 108)
I20260928 22:56:05.371822 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 27319
I20260928 22:56:05.372550 27615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:05.371912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:34361 (local address 127.0.0.1:57988)
0928 22:56:05.372484 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:05.372521 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:34361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552}
W20260928 22:56:05.372733 27103 catalog_manager.cc:4584] TS 072ef99cb67549efa45e527cd16d6479 (127.14.14.129:34361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82f00363bca34f99b49921f701c575fa: Network error: Client connection negotiation failed: client connection to 127.14.14.129:34361: connect: Connection refused (error 111)
I20260928 22:56:05.377534 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 27454
I20260928 22:56:05.383100 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 27086
I20260928 22:56:05.390718 14394 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 22:56:05.735462 14394 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 22:56:05.735580 14394 mini_ranger.cc:66] Stopping Ranger...
I20260928 22:56:06.079967 14394 mini_ranger.cc:68] Stopped Ranger
2026-09-28 22:56:06.080 UTC [26153] LOG: received smart shutdown request
2026-09-28 22:56:06.084 UTC [26153] LOG: background worker "logical replication launcher" (PID 26162) exited with exit code 1
2026-09-28 22:56:06.084 UTC [26157] LOG: shutting down
2026-09-28 22:56:06.274 UTC [26153] LOG: database system is shut down
2026-09-28T22:56:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47874 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:56:06 dist-test-slave-kzfr krb5kdc[27663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:56:06 dist-test-slave-kzfr krb5kdc[27663](info): set up 2 sockets
Sep 28 22:56:06 dist-test-slave-kzfr krb5kdc[27663](info): commencing operation
krb5kdc: starting...
W20260928 22:56:07.449875 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:56:07 dist-test-slave-kzfr krb5kdc[27663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:56:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:56:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:56:07.573506 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:39883
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:39883,127.14.14.189:35097,127.14.14.188:37447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:07.655488 27676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:07.655761 27676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:07.655802 27676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:07.656965 27676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:56:07.657037 27676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:07.657061 27676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:07.657090 27676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:56:07.657120 27676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:56:07.658751 27676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:39883,127.14.14.189:35097,127.14.14.188:37447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:39883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:07.659031 27676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:07.659211 27676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:07.664105 27682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:07.664112 27681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:07.664167 27676 server_base.cc:1034] running on GCE node
W20260928 22:56:07.664116 27684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:07.664590 27676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:07.664894 27676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:07.666065 27676 hybrid_clock.cc:648] HybridClock initialized: now 1790636167666049 us; error 47 us; skew 500 ppm
Sep 28 22:56:07 dist-test-slave-kzfr krb5kdc[27663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:07.668860 27676 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:56:07.669404 27676 webserver.cc:460] Webserver started at http://127.14.14.190:36523/ using document root <none> and password file <none>
I20260928 22:56:07.669697 27676 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:07.669767 27676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:07.669929 27676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:56:07.671041 27676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "09b11964dfd64b2ab746fdef26864a93"
format_stamp: "Formatted at 2026-09-28 22:56:07 on dist-test-slave-kzfr"
server_key: "8885013c45c160b9148b0e02e108f290"
server_key_iv: "5e88026ebfc20471a293276a37071046"
server_key_version: "encryptionkey@0"
I20260928 22:56:07.671466 27676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "09b11964dfd64b2ab746fdef26864a93"
format_stamp: "Formatted at 2026-09-28 22:56:07 on dist-test-slave-kzfr"
server_key: "8885013c45c160b9148b0e02e108f290"
server_key_iv: "5e88026ebfc20471a293276a37071046"
server_key_version: "encryptionkey@0"
I20260928 22:56:07.672811 27676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:56:07.673820 27691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:07.674038 27676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:56:07.674151 27676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "09b11964dfd64b2ab746fdef26864a93"
format_stamp: "Formatted at 2026-09-28 22:56:07 on dist-test-slave-kzfr"
server_key: "8885013c45c160b9148b0e02e108f290"
server_key_iv: "5e88026ebfc20471a293276a37071046"
server_key_version: "encryptionkey@0"
I20260928 22:56:07.674320 27676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:07.687674 27676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:07.690094 27676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:07.690286 27676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:07.694510 27676 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:39883
I20260928 22:56:07.694547 27743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:39883 every 8 connection(s)
I20260928 22:56:07.694931 27676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:56:07.695834 27744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:07.697407 27744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:56:07.698352 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 27676
I20260928 22:56:07.698462 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:56:07.698652 14394 external_mini_cluster.cc:1428] Setting key a2af2b166feb4a933ea12428cb22d8ba
Sep 28 22:56:07 dist-test-slave-kzfr krb5kdc[27663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.14.189@KRBTEST.COM; defaulting to no policy
I20260928 22:56:07.708966 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.698146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:39883 (local address 127.0.0.1:50818)
0928 22:56:07.698354 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.698401 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:07.699075 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:07.699404 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:07.699407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:07.699798 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:07.700039 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:07.700046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.701635 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.701636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:07.702815 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:07.702818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.703313 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.703314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:07.703412 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:07.703955 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:07.703963 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:07.705309 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:07.707351 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:07.707357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:07.707359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:07.707543 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:07.707900 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:07.707901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:07.707901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:07.707990 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:07.708515 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:07.708521 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:07.708580 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 22:56:07.708662 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 22:56:07.708887 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":82,"thread_start_us":83,"threads_started":1}
I20260928 22:56:07.708971 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.698314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.14.14.190:39883)
0928 22:56:07.698436 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:07.698441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:07.698483 (+ 42us) server_negotiation.cc:407] Connection header received
0928 22:56:07.699255 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:07.699258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:07.699275 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:07.699367 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:07.700174 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.701497 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.702899 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.703205 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.703338 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:07.705398 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:07.705406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:07.705406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:07.705414 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:07.707272 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:07.707720 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:07.707723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:07.707723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:07.707805 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:07.708169 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:07.708170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:07.708171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:07.708419 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:07.708501 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:07.708705 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 22:56:07.708798 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.189@KRBTEST.COM" created.
I20260928 22:56:07.709838 27744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:56:07.710186 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.710067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:35097 (local address 127.0.0.1:37418)
0928 22:56:07.710117 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.710159 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 22:56:07.710359 27694 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.14.189:35097: connect: Connection refused (error 111)
I20260928 22:56:07.710786 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.710697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:35097 (local address 127.0.0.1:37422)
0928 22:56:07.710746 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.710770 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 22:56:07.710925 27744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.189:35097: Network error: Client connection negotiation failed: client connection to 127.14.14.189:35097: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.189.keytab.
Entry for principal kudu/127.14.14.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.189.keytab.
I20260928 22:56:07.725504 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.189:35097
--webserver_interface=127.14.14.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.189.keytab
--principal=kudu/127.14.14.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:39883,127.14.14.189:35097,127.14.14.188:37447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:56:07.760183 27744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } attempt: 1
I20260928 22:56:07.760790 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.760593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:35097 (local address 127.0.0.1:37426)
0928 22:56:07.760678 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.760743 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 22:56:07.761215 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.761081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:35097 (local address 127.0.0.1:37428)
0928 22:56:07.761139 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.761183 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 22:56:07.761351 27744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.189:35097: Network error: Client connection negotiation failed: client connection to 127.14.14.189:35097: connect: Connection refused (error 111)
W20260928 22:56:07.810369 27751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:07.810590 27751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:07.810645 27751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:07.812490 27751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:56:07.812538 27751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:07.812553 27751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:07.812566 27751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:56:07.812579 27751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:56:07.814160 27751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:39883,127.14.14.189:35097,127.14.14.188:37447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.189.keytab
--principal=kudu/127.14.14.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.189:35097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.14.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:07.814360 27751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:07.814611 27751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:07.819530 27757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:07.819543 27756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:07.819664 27751 server_base.cc:1034] running on GCE node
W20260928 22:56:07.819525 27759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:07.819967 27751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:07.820343 27751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:07.821527 27751 hybrid_clock.cc:648] HybridClock initialized: now 1790636167821511 us; error 28 us; skew 500 ppm
Sep 28 22:56:07 dist-test-slave-kzfr krb5kdc[27663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:07.824259 27751 init.cc:377] Logged in from keytab as kudu/127.14.14.189@KRBTEST.COM (short username kudu)
I20260928 22:56:07.824738 27751 webserver.cc:460] Webserver started at http://127.14.14.189:45053/ using document root <none> and password file <none>
I20260928 22:56:07.824939 27751 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:07.824980 27751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:07.825071 27751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:56:07.826185 27751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/data/instance:
uuid: "2989f8f0249d40f08afc7bcd234200e3"
format_stamp: "Formatted at 2026-09-28 22:56:07 on dist-test-slave-kzfr"
server_key: "88dabbef1fd94f0dfaf354fa234923ad"
server_key_iv: "0ae400fb14a471f66853b675f44abf42"
server_key_version: "encryptionkey@0"
I20260928 22:56:07.826506 27751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/wal/instance:
uuid: "2989f8f0249d40f08afc7bcd234200e3"
format_stamp: "Formatted at 2026-09-28 22:56:07 on dist-test-slave-kzfr"
server_key: "88dabbef1fd94f0dfaf354fa234923ad"
server_key_iv: "0ae400fb14a471f66853b675f44abf42"
server_key_version: "encryptionkey@0"
I20260928 22:56:07.827752 27751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:56:07.828474 27766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:07.828657 27751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:07.828766 27751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/wal
uuid: "2989f8f0249d40f08afc7bcd234200e3"
format_stamp: "Formatted at 2026-09-28 22:56:07 on dist-test-slave-kzfr"
server_key: "88dabbef1fd94f0dfaf354fa234923ad"
server_key_iv: "0ae400fb14a471f66853b675f44abf42"
server_key_version: "encryptionkey@0"
I20260928 22:56:07.828846 27751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:07.829573 27744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } attempt: 2
I20260928 22:56:07.830034 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.829890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:35097 (local address 127.0.0.1:37432)
0928 22:56:07.829952 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.829994 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:56:07.830435 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.830338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:35097 (local address 127.0.0.1:37440)
0928 22:56:07.830383 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.830411 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 22:56:07.830605 27744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.189:35097: Network error: Client connection negotiation failed: client connection to 127.14.14.189:35097: connect: Connection refused (error 111)
I20260928 22:56:07.845287 27751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:07.848172 27751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:07.848363 27751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:07.852741 27751 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.189:35097
I20260928 22:56:07.852779 27818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.189:35097 every 8 connection(s)
I20260928 22:56:07.853137 27751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/data/info.pb
I20260928 22:56:07.854063 27819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:07.855554 27819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:56:07.860100 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 27751
I20260928 22:56:07.860213 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-1/wal/instance
I20260928 22:56:07.860476 14394 external_mini_cluster.cc:1428] Setting key a2f091c535f36527d0d97ed009630987
Sep 28 22:56:07 dist-test-slave-kzfr krb5kdc[27663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.189@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:07.865142 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.856255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50832 (local address 127.14.14.190:39883)
0928 22:56:07.856341 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:07.856343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:07.856528 (+ 185us) server_negotiation.cc:407] Connection header received
0928 22:56:07.856975 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:07.856976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:07.856987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:07.857048 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:07.857668 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.858150 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.859090 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.859407 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.859465 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:07.862106 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:07.862108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:07.862109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:07.862117 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:07.863564 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:07.864071 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:07.864072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:07.864072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:07.864125 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:07.864411 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:07.864412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:07.864412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:07.864578 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:07.864633 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:07.865010 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 22:56:07.865077 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 22:56:07.865465 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.856228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:39883 (local address 127.0.0.1:50832)
0928 22:56:07.856389 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.856432 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:07.856896 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:07.857109 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:07.857113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:07.857358 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:07.857550 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:07.857554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.858227 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.858228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:07.859003 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:07.859005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.859482 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.859483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:07.859579 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:07.860076 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:07.860084 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:07.861998 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:07.863652 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:07.863660 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:07.863662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:07.863993 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:07.864194 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:07.864195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:07.864195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:07.864344 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:07.864650 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:07.864657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:07.864863 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 22:56:07.864991 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 22:56:07.865196 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":118,"threads_started":1}
I20260928 22:56:07.866135 27819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.14.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.188@KRBTEST.COM" created.
Sep 28 22:56:07 dist-test-slave-kzfr krb5kdc[27663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.189@KRBTEST.COM for kudu/127.14.14.189@KRBTEST.COM
I20260928 22:56:07.876247 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.866506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.14.14.189:35097)
0928 22:56:07.866663 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:07.866667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:07.866683 (+ 16us) server_negotiation.cc:407] Connection header received
0928 22:56:07.866699 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:07.866701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:07.866711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:07.866788 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:07.867061 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.867767 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.868442 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.868801 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.868891 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:07.872861 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:07.872871 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:07.872872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:07.872881 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:07.874812 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:07.875238 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:07.875242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:07.875242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:07.875318 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:07.875573 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:07.875574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:07.875574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:07.875728 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:07.875799 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:07.876022 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 22:56:07.876113 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260928 22:56:07.876257 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.866435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:35097 (local address 127.0.0.1:37448)
0928 22:56:07.866481 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.866491 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:07.866534 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:07.866788 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:07.866789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:07.866944 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:07.866987 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:07.866988 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.867857 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.867858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:07.868349 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:07.868350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.868892 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.868892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:07.868958 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:07.869336 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:07.869341 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:07.872733 (+ 3392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:07.874898 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:07.874899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:07.874899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:07.875134 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:07.875397 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:07.875398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:07.875398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:07.875470 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:07.875808 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:07.875809 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:07.875862 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 22:56:07.875942 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 22:56:07.876050 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 22:56:07.877018 27819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:56:07.877519 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.877398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:37447 (local address 127.0.0.1:56340)
0928 22:56:07.877454 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.877500 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 22:56:07.877699 27768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
I20260928 22:56:07.878159 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.878077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:37447 (local address 127.0.0.1:56342)
0928 22:56:07.878120 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.878143 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:56:07.878275 27819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.188:37447: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.188.keytab.
Entry for principal kudu/127.14.14.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.188.keytab.
I20260928 22:56:07.889747 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.188:37447
--webserver_interface=127.14.14.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.188.keytab
--principal=kudu/127.14.14.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:39883,127.14.14.189:35097,127.14.14.188:37447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:56:07.921754 27744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } attempt: 3
Sep 28 22:56:07 dist-test-slave-kzfr krb5kdc[27663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for kudu/127.14.14.189@KRBTEST.COM
I20260928 22:56:07.927507 27819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } attempt: 1
I20260928 22:56:07.927881 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.927764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:37447 (local address 127.0.0.1:56356)
0928 22:56:07.927817 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.927853 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 22:56:07.928234 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.928131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:37447 (local address 127.0.0.1:56372)
0928 22:56:07.928175 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.928201 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 22:56:07.928342 27819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.188:37447: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
I20260928 22:56:07.929346 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.922126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:35097 (local address 127.0.0.1:37456)
0928 22:56:07.922193 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.922210 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:07.922275 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:07.922456 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:07.922457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:07.922618 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:07.922686 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:07.922688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.923456 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.923462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:07.923984 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:07.923986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.924379 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.924380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:07.924463 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:07.924935 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:07.924940 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:07.926481 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:07.928434 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:07.928435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:07.928436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:07.928601 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:07.928803 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:07.928804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:07.928804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:07.928870 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:07.929121 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:07.929121 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:07.929158 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 22:56:07.929209 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:56:07.929283 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 22:56:07.929418 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.922208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.14.14.189:35097)
0928 22:56:07.922275 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:07.922277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:07.922289 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:56:07.922366 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:07.922367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:07.922386 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:07.922449 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:07.922767 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.923367 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.924066 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:07.924285 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:07.924369 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:07.926570 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:07.926571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:07.926572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:07.926579 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:07.928341 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:07.928675 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:07.928676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:07.928677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:07.928738 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:07.928928 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:07.928929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:07.928930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:07.929059 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:07.929124 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:07.929258 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 22:56:07.929335 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:56:07.930054 27744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:56:07.930384 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.930285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:37447 (local address 127.0.0.1:56376)
0928 22:56:07.930331 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.930358 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 22:56:07.930671 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.930619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:37447 (local address 127.0.0.1:56384)
0928 22:56:07.930642 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.930660 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 22:56:07.930764 27744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.188:37447: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
I20260928 22:56:07.961895 27744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } attempt: 1
I20260928 22:56:07.962450 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.962313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:37447 (local address 127.0.0.1:56400)
0928 22:56:07.962381 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.962423 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 22:56:07.962849 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.962752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:37447 (local address 127.0.0.1:56408)
0928 22:56:07.962794 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.962819 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 22:56:07.962965 27744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.188:37447: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
W20260928 22:56:07.972054 27826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:07.972249 27826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:07.972271 27826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:07.973340 27826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:56:07.973376 27826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:07.973392 27826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:07.973403 27826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:56:07.973415 27826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:56:07.974913 27826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:39883,127.14.14.189:35097,127.14.14.188:37447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.188.keytab
--principal=kudu/127.14.14.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.188:37447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.14.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:07.975097 27826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:07.975260 27826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:07.980233 27834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:07.980208 27832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:07.980213 27826 server_base.cc:1034] running on GCE node
W20260928 22:56:07.980203 27831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:07.980688 27826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:07.980948 27826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:07.982121 27826 hybrid_clock.cc:648] HybridClock initialized: now 1790636167982106 us; error 26 us; skew 500 ppm
Sep 28 22:56:07 dist-test-slave-kzfr krb5kdc[27663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:07.984622 27826 init.cc:377] Logged in from keytab as kudu/127.14.14.188@KRBTEST.COM (short username kudu)
I20260928 22:56:07.985042 27826 webserver.cc:460] Webserver started at http://127.14.14.188:35363/ using document root <none> and password file <none>
I20260928 22:56:07.985284 27826 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:07.985353 27826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:07.985505 27826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:56:07.986655 27826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/data/instance:
uuid: "fcb6236babd9434e8e10f2171de14a16"
format_stamp: "Formatted at 2026-09-28 22:56:07 on dist-test-slave-kzfr"
server_key: "d6813488e9dc30ce155416593dbae72b"
server_key_iv: "70cc6fe772e660640c9dabb2b55a892d"
server_key_version: "encryptionkey@0"
I20260928 22:56:07.987013 27826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/wal/instance:
uuid: "fcb6236babd9434e8e10f2171de14a16"
format_stamp: "Formatted at 2026-09-28 22:56:07 on dist-test-slave-kzfr"
server_key: "d6813488e9dc30ce155416593dbae72b"
server_key_iv: "70cc6fe772e660640c9dabb2b55a892d"
server_key_version: "encryptionkey@0"
I20260928 22:56:07.988343 27826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:56:07.989078 27841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:07.989315 27826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:07.989418 27826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/wal
uuid: "fcb6236babd9434e8e10f2171de14a16"
format_stamp: "Formatted at 2026-09-28 22:56:07 on dist-test-slave-kzfr"
server_key: "d6813488e9dc30ce155416593dbae72b"
server_key_iv: "70cc6fe772e660640c9dabb2b55a892d"
server_key_version: "encryptionkey@0"
I20260928 22:56:07.989499 27826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:07.996488 27819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } attempt: 2
I20260928 22:56:07.996893 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.996764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:37447 (local address 127.0.0.1:56412)
0928 22:56:07.996825 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.996864 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 22:56:07.997198 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:07.997108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:37447 (local address 127.0.0.1:56422)
0928 22:56:07.997151 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:07.997175 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 22:56:07.997368 27819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.188:37447: Network error: Client connection negotiation failed: client connection to 127.14.14.188:37447: connect: Connection refused (error 111)
I20260928 22:56:08.002800 27826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:08.004897 27826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:08.005007 27826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:08.009509 27826 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.188:37447
I20260928 22:56:08.009546 27893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.188:37447 every 8 connection(s)
I20260928 22:56:08.009907 27826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/data/info.pb
I20260928 22:56:08.010811 27894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:08.012398 27894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:56:08.014320 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 27826
I20260928 22:56:08.014441 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790636016256058-14394-0/minicluster-data/master-2/wal/instance
I20260928 22:56:08.014649 14394 external_mini_cluster.cc:1428] Setting key fcab1ea2c3f61ae43f7e3c731790cd01
Sep 28 22:56:08 dist-test-slave-kzfr krb5kdc[27663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.188@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
Sep 28 22:56:08 dist-test-slave-kzfr krb5kdc[27663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.188@KRBTEST.COM
I20260928 22:56:08.022415 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.013151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.14.14.190:39883)
0928 22:56:08.013239 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:08.013241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:08.013387 (+ 146us) server_negotiation.cc:407] Connection header received
0928 22:56:08.013905 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:08.013906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:08.013916 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:08.013971 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:08.014658 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.015297 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.016421 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.016624 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.016676 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.018859 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:08.018860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:08.018860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:08.018868 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:08.020971 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.021413 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.021414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.021414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.021475 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.021762 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.021763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.021763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.021885 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:08.021957 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:08.022239 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 22:56:08.022323 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 22:56:08.022603 27896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.013059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:39883 (local address 127.0.0.1:50838)
0928 22:56:08.013221 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:08.013303 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:08.013814 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:08.014039 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:08.014042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:08.014290 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:08.014553 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:08.014558 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.015399 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.015400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:08.016339 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:08.016341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.016699 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.016700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:08.016844 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.017358 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:08.017367 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:08.018753 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:08.021064 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:08.021071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:08.021073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:08.021328 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:08.021544 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:08.021544 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:08.021545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:08.021691 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:08.021962 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:08.021969 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:08.022104 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 22:56:08.022205 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 22:56:08.022363 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
I20260928 22:56:08.023140 27894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:56:08.023784 27898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.015029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56428 (local address 127.14.14.188:37447)
0928 22:56:08.015450 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:08.015455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:08.015475 (+ 20us) server_negotiation.cc:407] Connection header received
0928 22:56:08.015493 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:08.015495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:08.015515 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:08.015589 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:08.016255 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.017177 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.018035 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.018329 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.018419 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.020646 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:08.020654 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:08.020654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:08.020661 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:08.022307 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.022746 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.022749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.022749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.022806 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.022999 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.022999 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.023000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.023179 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:08.023337 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:08.023448 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 22:56:08.023535 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":89,"threads_started":1}
Sep 28 22:56:08 dist-test-slave-kzfr krb5kdc[27663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.188@KRBTEST.COM for kudu/127.14.14.189@KRBTEST.COM
Sep 28 22:56:08 dist-test-slave-kzfr krb5kdc[27663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:08.030308 27896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.023418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:35097 (local address 127.0.0.1:37470)
0928 22:56:08.023515 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:08.023524 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:08.023559 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:08.023709 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:08.023710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:08.023856 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:08.023926 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:08.023928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.024690 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.024691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:08.025171 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:08.025173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.025564 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.025565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:08.025667 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.026074 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:08.026079 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:08.027166 (+ 1087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:08.029287 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:08.029288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:08.029288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:08.029453 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:08.029676 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:08.029677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:08.029677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:08.029734 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:08.029998 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:08.029999 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:08.030039 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 22:56:08.030119 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 22:56:08.030213 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 22:56:08.030314 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.023546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.14.14.189:35097)
0928 22:56:08.023593 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:08.023594 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:08.023602 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:56:08.023625 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:08.023626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:08.023635 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:08.023689 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:08.024007 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.024616 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.025255 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.025477 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.025563 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.027253 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:08.027254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:08.027255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:08.027263 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:08.029196 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.029526 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.029527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.029528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.029625 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.029805 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.029806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.029807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.029923 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:08.029994 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:08.030157 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 22:56:08.030230 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:56:08.030656 27894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:56:08.031600 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.024541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.14.14.190:39883)
0928 22:56:08.024573 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:08.024574 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:08.024582 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:56:08.024629 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:08.024630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:08.024639 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:08.024690 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:08.025205 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.025718 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.026380 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.026585 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.026667 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.028933 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:08.028934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:08.028934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:08.028940 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:08.030320 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.030720 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.030721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.030721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.030781 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.030997 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.030998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.030999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.031128 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:08.031366 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:08.031406 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 22:56:08.031485 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 22:56:08 dist-test-slave-kzfr krb5kdc[27663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.188@KRBTEST.COM for kudu/127.14.14.188@KRBTEST.COM
Sep 28 22:56:08 dist-test-slave-kzfr krb5kdc[27663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.189@KRBTEST.COM
I20260928 22:56:08.038067 27898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.031014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.14.14.188:37447)
0928 22:56:08.031052 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:08.031054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:08.031062 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:56:08.031090 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:08.031091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:08.031097 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:08.031168 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:08.031440 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.032065 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.032864 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.033071 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.033152 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.035354 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:08.035355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:08.035356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:08.035365 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:08.036745 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.037146 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.037147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.037148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.037204 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.037402 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.037403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.037404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.037551 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:08.037672 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:08.037864 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 22:56:08.037965 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":21}
I20260928 22:56:08.038113 27744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } attempt: 2
I20260928 22:56:08.038077 27896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.030953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:37447 (local address 127.0.0.1:56444)
0928 22:56:08.030997 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:08.031006 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:08.031038 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:08.031169 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:08.031170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:08.031302 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:08.031356 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:08.031358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.032142 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.032143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:08.032775 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:08.032777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.033153 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.033153 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:08.033202 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.033662 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:08.033667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:08.035134 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:08.036862 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:08.036863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:08.036863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:08.037064 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:08.037275 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:08.037276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:08.037277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:08.037330 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:08.037677 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:08.037678 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:08.037727 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 22:56:08.037816 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 22:56:08.037919 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 22:56:08.039784 27894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16: Bootstrap starting.
I20260928 22:56:08.040427 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.032253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37472 (local address 127.14.14.189:35097)
0928 22:56:08.032290 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:08.032292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:08.032299 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:56:08.032361 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:08.032362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:08.032370 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:08.032443 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:08.032930 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.033420 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.034577 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.034894 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.034977 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.037306 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:08.037307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:08.037308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:08.037316 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:08.039170 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.039555 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.039557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.039557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.039611 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.039788 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.039789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.039789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.039923 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:08.040055 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:08.040247 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 22:56:08.040327 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:56:08.040839 27894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:08.041175 27894 log.cc:824] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:08.042196 27894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16: No bootstrap required, opened a new log
I20260928 22:56:08.042266 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 28 22:56:08 dist-test-slave-kzfr krb5kdc[27663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for kudu/127.14.14.188@KRBTEST.COM
I20260928 22:56:08.044595 27894 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } }
I20260928 22:56:08.044768 27894 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:08.044809 27894 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb6236babd9434e8e10f2171de14a16, State: Initialized, Role: FOLLOWER
I20260928 22:56:08.044958 27894 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } }
I20260928 22:56:08.045288 27894 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:56:08.046314 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.038394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:37447 (local address 127.0.0.1:56452)
0928 22:56:08.038507 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:08.038517 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:08.038554 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:08.038770 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:08.038771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:08.038936 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:08.038981 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:08.038983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.039876 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.039877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:08.040648 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:08.040651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.041083 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.041085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:08.041167 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.041693 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:08.041698 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:08.043176 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:08.045053 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:08.045054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:08.045055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:08.045322 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:08.045532 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:08.045533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:08.045533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:08.045641 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:08.045986 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:08.045987 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:08.046037 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 22:56:08.046105 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 22:56:08.046199 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 22:56:08.046638 27898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.038449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.14.14.188:37447)
0928 22:56:08.038634 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:08.038636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:08.038644 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:56:08.038661 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:08.038662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:08.038671 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:08.038757 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:08.039061 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.039782 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.040732 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.041000 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.041085 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.043293 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:08.043294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:08.043294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:08.043302 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:08.044964 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.045400 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.045401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.045402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.045460 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.045717 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.045718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.045719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.045886 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:08.045956 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:08.046487 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 22:56:08.046558 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
W20260928 22:56:08.047726 27919 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:56:08.047845 27919 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:56:08.048185 27744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93: Bootstrap starting.
I20260928 22:56:08.048957 27744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:08.049280 27744 log.cc:824] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:08.051491 27744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93: No bootstrap required, opened a new log
I20260928 22:56:08.053920 27744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } }
I20260928 22:56:08.050366 27826 master_runner.cc:186] Error getting master registration for 127.14.14.190:39883: OK, instance_id { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" instance_seqno: 1790636167693298 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:56:08.054116 27826 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:56:08.054148 27744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:08.054239 27744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b11964dfd64b2ab746fdef26864a93, State: Initialized, Role: FOLLOWER
I20260928 22:56:08.054443 27744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } }
I20260928 22:56:08.054625 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.043938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.14.14.190:39883)
0928 22:56:08.043982 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:08.043984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:08.044157 (+ 173us) server_negotiation.cc:407] Connection header received
0928 22:56:08.044541 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:08.044542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:08.044551 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:08.044580 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:08.046973 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.047580 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.049827 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.050113 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.050173 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.051523 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:08.051525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:08.051525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:08.051532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:08.053186 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.053560 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.053561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.053561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.053645 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.053854 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.053855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.053856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.054006 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:08.054080 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:08.054467 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 22:56:08.054544 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:56:08.054908 27908 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } } }
I20260928 22:56:08.055014 27744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:56:08.055327 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.043972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37486 (local address 127.14.14.189:35097)
0928 22:56:08.044008 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:08.044009 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:08.044394 (+ 385us) server_negotiation.cc:407] Connection header received
0928 22:56:08.044492 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:08.044493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:08.044502 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:08.044532 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:08.045257 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.046055 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.047169 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.047426 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.047468 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.049022 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:08.049024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:08.049024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:08.049031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:08.050809 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.052449 (+ 1640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.052450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.052450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.052509 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.054413 (+ 1904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.054415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.054415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.054566 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:08.054649 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:08.055110 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 22:56:08.055206 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:56:08.055459 27908 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:56:08.056957 27933 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:56:08.057047 27933 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:56:08.057492 27922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } } }
I20260928 22:56:08.057657 27922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:56:08.057814 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.043812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56468 (local address 127.14.14.188:37447)
0928 22:56:08.044041 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:08.044043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:08.044245 (+ 202us) server_negotiation.cc:407] Connection header received
0928 22:56:08.044330 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:08.044331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:08.044343 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:08.044405 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:08.046004 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.046806 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.048502 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.048815 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.048878 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.054496 (+ 5618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:08.054497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:08.054497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:08.054504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:08.056310 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.056682 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.056684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.056684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.056743 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.056962 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.056963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.056963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.057136 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:08.057248 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:08.057562 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 22:56:08.057686 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
I20260928 22:56:08.057529 27676 master_runner.cc:186] Error getting master registration for 127.14.14.189:35097: OK, instance_id { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" instance_seqno: 1790636167851455 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:56:08.060356 27676 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:56:08.088546 27819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } attempt: 3
Sep 28 22:56:08 dist-test-slave-kzfr krb5kdc[27663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636167, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.189@KRBTEST.COM for kudu/127.14.14.188@KRBTEST.COM
I20260928 22:56:08.097221 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.088872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:37447 (local address 127.0.0.1:56474)
0928 22:56:08.088924 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:08.088936 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:08.088975 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:08.089124 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:08.089125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:08.089316 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:08.089369 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:08.089371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.090349 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.090350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:08.091165 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:08.091167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.091699 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.091700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:08.091760 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.092551 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:08.092556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:08.094214 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:08.096085 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:08.096086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:08.096086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:08.096317 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:08.096545 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:08.096546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:08.096547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:08.096614 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:08.096897 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:08.096898 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:08.096946 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 22:56:08.097019 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:56:08.097116 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 22:56:08.097232 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:08.088917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.14.14.188:37447)
0928 22:56:08.088958 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:08.088960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:08.088969 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:56:08.089029 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:08.089029 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:08.089040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:08.089105 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:08.089447 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.090251 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.091256 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:08.091627 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:08.091694 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:08.094304 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:08.094305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:08.094306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:08.094312 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:08.095998 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.096397 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.096398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.096398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.096472 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:08.096684 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:08.096684 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:08.096685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:08.096818 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:08.096885 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:08.097054 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 22:56:08.097137 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:56:08.099258 27819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3: Bootstrap starting.
I20260928 22:56:08.100028 27819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:08.100327 27819 log.cc:824] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:08.101227 27819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3: No bootstrap required, opened a new log
I20260928 22:56:08.103292 27819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } }
I20260928 22:56:08.103886 27819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:08.103994 27819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2989f8f0249d40f08afc7bcd234200e3, State: Initialized, Role: FOLLOWER
I20260928 22:56:08.104148 27819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } }
I20260928 22:56:08.104425 27936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } } }
I20260928 22:56:08.104557 27936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:56:08.104480 27819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 22:56:08.106637 27947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:56:08.106740 27947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:56:08.108465 27751 master_runner.cc:263] No leader master found from master 2989f8f0249d40f08afc7bcd234200e3
I20260928 22:56:08.108511 27751 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 22:56:08.191104 27697 tablet.cc:2369] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:56:08.255618 27847 tablet.cc:2369] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:56:08.267807 27936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:56:08.267941 27936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } }
I20260928 22:56:08.268239 27936 leader_election.cc:288] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09b11964dfd64b2ab746fdef26864a93 (127.14.14.190:39883), fcb6236babd9434e8e10f2171de14a16 (127.14.14.188:37447)
I20260928 22:56:08.268535 27718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2989f8f0249d40f08afc7bcd234200e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09b11964dfd64b2ab746fdef26864a93" is_pre_election: true
I20260928 22:56:08.268541 27868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2989f8f0249d40f08afc7bcd234200e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb6236babd9434e8e10f2171de14a16" is_pre_election: true
I20260928 22:56:08.268713 27868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2989f8f0249d40f08afc7bcd234200e3 in term 0.
I20260928 22:56:08.268707 27718 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2989f8f0249d40f08afc7bcd234200e3 in term 0.
I20260928 22:56:08.268922 27768 leader_election.cc:302] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2989f8f0249d40f08afc7bcd234200e3, fcb6236babd9434e8e10f2171de14a16; no voters:
I20260928 22:56:08.269064 27936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:56:08.269152 27936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:56:08.269205 27936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:08.270229 27936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } }
I20260928 22:56:08.270546 27718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2989f8f0249d40f08afc7bcd234200e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09b11964dfd64b2ab746fdef26864a93"
I20260928 22:56:08.270601 27868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2989f8f0249d40f08afc7bcd234200e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb6236babd9434e8e10f2171de14a16"
I20260928 22:56:08.270646 27718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:08.270679 27868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:08.271338 27868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2989f8f0249d40f08afc7bcd234200e3 in term 1.
I20260928 22:56:08.271526 27768 leader_election.cc:302] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2989f8f0249d40f08afc7bcd234200e3, fcb6236babd9434e8e10f2171de14a16; no voters:
I20260928 22:56:08.271553 27936 leader_election.cc:288] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [CANDIDATE]: Term 1 election: Requested vote from peers 09b11964dfd64b2ab746fdef26864a93 (127.14.14.190:39883), fcb6236babd9434e8e10f2171de14a16 (127.14.14.188:37447)
I20260928 22:56:08.271550 27718 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2989f8f0249d40f08afc7bcd234200e3 in term 1.
I20260928 22:56:08.271596 27936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:56:08.271636 27936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [term 1 LEADER]: Becoming Leader. State: Replica: 2989f8f0249d40f08afc7bcd234200e3, State: Running, Role: LEADER
I20260928 22:56:08.271766 27936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } }
I20260928 22:56:08.272245 27949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2989f8f0249d40f08afc7bcd234200e3. Latest consensus state: current_term: 1 leader_uuid: "2989f8f0249d40f08afc7bcd234200e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } } }
I20260928 22:56:08.272334 27949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:08.272528 27951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:56:08.272713 27951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:56:08.274045 27718 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93 [term 1 FOLLOWER]: Refusing update from remote peer 2989f8f0249d40f08afc7bcd234200e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:56:08.274082 27868 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16 [term 1 FOLLOWER]: Refusing update from remote peer 2989f8f0249d40f08afc7bcd234200e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:56:08.274353 27949 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:56:08.274453 27936 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:56:08.275607 27908 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2989f8f0249d40f08afc7bcd234200e3. Latest consensus state: current_term: 1 leader_uuid: "2989f8f0249d40f08afc7bcd234200e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } } }
I20260928 22:56:08.275713 27908 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:56:08.276057 27922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2989f8f0249d40f08afc7bcd234200e3. Latest consensus state: current_term: 1 leader_uuid: "2989f8f0249d40f08afc7bcd234200e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } } }
I20260928 22:56:08.276140 27922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:56:08.276412 27922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2989f8f0249d40f08afc7bcd234200e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } } }
I20260928 22:56:08.276481 27922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:56:08.276446 27936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2989f8f0249d40f08afc7bcd234200e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } } }
I20260928 22:56:08.276511 27936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:08.276696 27954 mvcc.cc:205] Tried to move back new op lower bound from 7334445745248112640 to 7334445745241853952. Current Snapshot: MvccSnapshot[applied={T|T < 7334445745248112640}]
I20260928 22:56:08.277009 27951 catalog_manager.cc:1347] Generated new cluster ID: 9660906bb4714da5894367c3c2837f53
I20260928 22:56:08.277083 27951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:56:08.281944 27936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2989f8f0249d40f08afc7bcd234200e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } } }
I20260928 22:56:08.282043 27936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:08.282366 27908 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2989f8f0249d40f08afc7bcd234200e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b11964dfd64b2ab746fdef26864a93" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 39883 } } peers { permanent_uuid: "2989f8f0249d40f08afc7bcd234200e3" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 35097 } } peers { permanent_uuid: "fcb6236babd9434e8e10f2171de14a16" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 37447 } } }
I20260928 22:56:08.282449 27908 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:56:08.285022 27951 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:56:08.285631 27951 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:56:08.292393 27951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2989f8f0249d40f08afc7bcd234200e3: Generated new TSK 0
I20260928 22:56:08.292795 27951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:56:09.048372 27919 catalog_manager.cc:1259] Loaded cluster ID: 9660906bb4714da5894367c3c2837f53
I20260928 22:56:09.048457 27919 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16: loading cluster ID for follower catalog manager: success
I20260928 22:56:09.049299 27919 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16: acquiring CA information for follower catalog manager: success
I20260928 22:56:09.049685 27919 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fcb6236babd9434e8e10f2171de14a16: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:56:09.057554 27933 catalog_manager.cc:1259] Loaded cluster ID: 9660906bb4714da5894367c3c2837f53
I20260928 22:56:09.057673 27933 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93: loading cluster ID for follower catalog manager: success
I20260928 22:56:09.058456 27933 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93: acquiring CA information for follower catalog manager: success
I20260928 22:56:09.058743 27933 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09b11964dfd64b2ab746fdef26864a93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:56:09.341355 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 27676
I20260928 22:56:09.346829 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 27751
I20260928 22:56:09.351576 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 27826
2026-09-28T22:56:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2956 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:56:09 dist-test-slave-kzfr krb5kdc[27966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:56:09 dist-test-slave-kzfr krb5kdc[27966](info): set up 2 sockets
Sep 28 22:56:09 dist-test-slave-kzfr krb5kdc[27966](info): commencing operation
krb5kdc: starting...
W20260928 22:56:10.413875 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:56:10 dist-test-slave-kzfr krb5kdc[27966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:56:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:56:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:56:10.532167 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:43233
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:43233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:10.614221 27979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:10.614429 27979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:10.614451 27979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:10.615623 27979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:56:10.615665 27979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:10.615679 27979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:10.615733 27979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:56:10.615759 27979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:56:10.617199 27979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:43233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:43233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:10.617456 27979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:10.617713 27979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:10.622617 27985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:10.622627 27987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:10.622627 27984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:10.622926 27979 server_base.cc:1034] running on GCE node
I20260928 22:56:10.623164 27979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:10.623435 27979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:10.624576 27979 hybrid_clock.cc:648] HybridClock initialized: now 1790636170624558 us; error 34 us; skew 500 ppm
Sep 28 22:56:10 dist-test-slave-kzfr krb5kdc[27966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636170, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:10.627060 27979 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:56:10.627580 27979 webserver.cc:460] Webserver started at http://127.14.14.190:43861/ using document root <none> and password file <none>
I20260928 22:56:10.627815 27979 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:10.627888 27979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:10.628039 27979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:56:10.629079 27979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "4ab063ac8d9848369f0f27f985b4b617"
format_stamp: "Formatted at 2026-09-28 22:56:10 on dist-test-slave-kzfr"
server_key: "1a80d6f2ab98da58b988266deb355574"
server_key_iv: "be32b37bd2f8f6f3ae1c4a209e12dab9"
server_key_version: "encryptionkey@0"
I20260928 22:56:10.629441 27979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "4ab063ac8d9848369f0f27f985b4b617"
format_stamp: "Formatted at 2026-09-28 22:56:10 on dist-test-slave-kzfr"
server_key: "1a80d6f2ab98da58b988266deb355574"
server_key_iv: "be32b37bd2f8f6f3ae1c4a209e12dab9"
server_key_version: "encryptionkey@0"
I20260928 22:56:10.630775 27979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:56:10.631511 27994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:10.631700 27979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:56:10.631789 27979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "4ab063ac8d9848369f0f27f985b4b617"
format_stamp: "Formatted at 2026-09-28 22:56:10 on dist-test-slave-kzfr"
server_key: "1a80d6f2ab98da58b988266deb355574"
server_key_iv: "be32b37bd2f8f6f3ae1c4a209e12dab9"
server_key_version: "encryptionkey@0"
I20260928 22:56:10.631873 27979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:10.644201 27979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:10.646085 27979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:10.646253 27979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:10.650337 27979 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:43233
I20260928 22:56:10.650369 28046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:43233 every 8 connection(s)
I20260928 22:56:10.650715 27979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:56:10.651615 28047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:10.653861 28047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617: Bootstrap starting.
I20260928 22:56:10.654388 28047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:10.654618 28047 log.cc:824] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:10.655274 28047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617: No bootstrap required, opened a new log
I20260928 22:56:10.656785 28047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab063ac8d9848369f0f27f985b4b617" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 43233 } }
I20260928 22:56:10.657001 28047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:10.657021 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 27979
I20260928 22:56:10.657042 28047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab063ac8d9848369f0f27f985b4b617, State: Initialized, Role: FOLLOWER
I20260928 22:56:10.657135 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:56:10.657194 28047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab063ac8d9848369f0f27f985b4b617" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 43233 } }
I20260928 22:56:10.657281 28047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:56:10.657313 28047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:56:10.657351 14394 external_mini_cluster.cc:1428] Setting key 30aafcd881b2f07293a20c47c11f7f5e
I20260928 22:56:10.657379 28047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:10.658389 28047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab063ac8d9848369f0f27f985b4b617" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 43233 } }
I20260928 22:56:10.658514 28047 leader_election.cc:302] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ab063ac8d9848369f0f27f985b4b617; no voters:
I20260928 22:56:10.658702 28047 leader_election.cc:288] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:56:10.658740 28052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:56:10.658870 28052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [term 1 LEADER]: Becoming Leader. State: Replica: 4ab063ac8d9848369f0f27f985b4b617, State: Running, Role: LEADER
I20260928 22:56:10.658969 28047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:56:10.658960 28052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab063ac8d9848369f0f27f985b4b617" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 43233 } }
I20260928 22:56:10.660226 28054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ab063ac8d9848369f0f27f985b4b617. Latest consensus state: current_term: 1 leader_uuid: "4ab063ac8d9848369f0f27f985b4b617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab063ac8d9848369f0f27f985b4b617" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 43233 } } }
I20260928 22:56:10.660363 28054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:10.660871 28053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ab063ac8d9848369f0f27f985b4b617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab063ac8d9848369f0f27f985b4b617" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 43233 } } }
I20260928 22:56:10.660948 28053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617 [sys.catalog]: This master's current role is: LEADER
W20260928 22:56:10.661010 28068 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:56:10.661167 28068 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:56:10.661260 28067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:56:10.661433 28067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:56:10.663487 28067 catalog_manager.cc:1347] Generated new cluster ID: fdde2f438946466c94d003ef3072bced
I20260928 22:56:10.663542 28067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:56:10 dist-test-slave-kzfr krb5kdc[27966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:10.669413 28051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:10.658309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.14.14.190:43233)
0928 22:56:10.658465 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:10.658475 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:10.658510 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:56:10.658647 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:10.658654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:10.658734 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:10.658882 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:10.659813 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:10.661314 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:10.662304 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:10.662927 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:10.663040 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:10.665726 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:10.665737 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:10.665738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:10.665750 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:10.667848 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:10.668238 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:10.668241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:10.668242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:10.668342 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:10.668570 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:10.668571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:10.668571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:10.668808 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:10.668998 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:10.669065 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 22:56:10.669204 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260928 22:56:10.670625 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:56:10.676936 28067 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:56:10.677486 28067 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:56:10.682535 28067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ab063ac8d9848369f0f27f985b4b617: Generated new TSK 0
I20260928 22:56:10.682715 28067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:56:10.724514 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 27979
2026-09-28T22:56:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1371 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:56:10 dist-test-slave-kzfr krb5kdc[28074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:56:10 dist-test-slave-kzfr krb5kdc[28074](info): set up 2 sockets
Sep 28 22:56:10 dist-test-slave-kzfr krb5kdc[28074](info): commencing operation
krb5kdc: starting...
W20260928 22:56:11.793943 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:56:11 dist-test-slave-kzfr krb5kdc[28074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:56:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:56:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:56:11.920744 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44663
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:44663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:12.002131 28087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:12.002339 28087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:12.002362 28087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:12.003463 28087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:56:12.003512 28087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:12.003527 28087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:12.003539 28087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:56:12.003552 28087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:56:12.004956 28087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:44663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:12.005144 28087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:12.005312 28087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:12.010206 28095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:12.010208 28093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:12.010154 28092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:12.010473 28087 server_base.cc:1034] running on GCE node
I20260928 22:56:12.010716 28087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:12.011013 28087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:12.012192 28087 hybrid_clock.cc:648] HybridClock initialized: now 1790636172012179 us; error 46 us; skew 500 ppm
Sep 28 22:56:12 dist-test-slave-kzfr krb5kdc[28074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:12.018549 28087 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:56:12.018940 28087 webserver.cc:460] Webserver started at http://127.14.14.190:44509/ using document root <none> and password file <none>
I20260928 22:56:12.019133 28087 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:12.019172 28087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:12.019260 28087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:56:12.020260 28087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "d89d2420acf6402cbba1f9fa04f70fa0"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "a1191e486fd9c8c723cc139fb1d9090e"
server_key_iv: "299d5b1ef61c70873ba3e2b90ebfaec2"
server_key_version: "encryptionkey@0"
I20260928 22:56:12.020560 28087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "d89d2420acf6402cbba1f9fa04f70fa0"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "a1191e486fd9c8c723cc139fb1d9090e"
server_key_iv: "299d5b1ef61c70873ba3e2b90ebfaec2"
server_key_version: "encryptionkey@0"
I20260928 22:56:12.021889 28087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:56:12.022579 28102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:12.022758 28087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:56:12.022845 28087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "d89d2420acf6402cbba1f9fa04f70fa0"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "a1191e486fd9c8c723cc139fb1d9090e"
server_key_iv: "299d5b1ef61c70873ba3e2b90ebfaec2"
server_key_version: "encryptionkey@0"
I20260928 22:56:12.022905 28087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:12.057546 28087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:12.059679 28087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:12.059793 28087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:12.063694 28087 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:44663
I20260928 22:56:12.063781 28154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:44663 every 8 connection(s)
I20260928 22:56:12.064139 28087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:56:12.065177 28155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:12.065719 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 28087
I20260928 22:56:12.065827 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:56:12.066022 14394 external_mini_cluster.cc:1428] Setting key 8b33346245f3e2ed09e639b59bf32324
I20260928 22:56:12.067972 28155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: Bootstrap starting.
I20260928 22:56:12.068667 28155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:12.068969 28155 log.cc:824] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:12.069725 28155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: No bootstrap required, opened a new log
I20260928 22:56:12.071825 28155 raft_consensus.cc:348] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } }
I20260928 22:56:12.071966 28155 raft_consensus.cc:374] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:12.071990 28155 raft_consensus.cc:729] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d89d2420acf6402cbba1f9fa04f70fa0, State: Initialized, Role: FOLLOWER
I20260928 22:56:12.072129 28155 consensus_queue.cc:260] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } }
I20260928 22:56:12.072201 28155 raft_consensus.cc:388] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:56:12.072223 28155 raft_consensus.cc:482] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:56:12.072292 28155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:12.072834 28155 raft_consensus.cc:504] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } }
I20260928 22:56:12.072968 28155 leader_election.cc:302] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d89d2420acf6402cbba1f9fa04f70fa0; no voters:
I20260928 22:56:12.073160 28155 leader_election.cc:288] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:56:12.073190 28160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 1 FOLLOWER]: Leader election won for term 1
Sep 28 22:56:12 dist-test-slave-kzfr krb5kdc[28074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:12.073316 28160 raft_consensus.cc:686] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 1 LEADER]: Becoming Leader. State: Replica: d89d2420acf6402cbba1f9fa04f70fa0, State: Running, Role: LEADER
I20260928 22:56:12.073405 28160 consensus_queue.cc:237] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } }
I20260928 22:56:12.073431 28155 sys_catalog.cc:564] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:56:12.073866 28162 sys_catalog.cc:455] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } } }
I20260928 22:56:12.074004 28162 sys_catalog.cc:458] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:12.073860 28160 sys_catalog.cc:455] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d89d2420acf6402cbba1f9fa04f70fa0. Latest consensus state: current_term: 1 leader_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } } }
I20260928 22:56:12.074489 28160 sys_catalog.cc:458] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:12.074600 28169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:56:12.075131 28169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:56:12.077227 28169 catalog_manager.cc:1347] Generated new cluster ID: 84e1c2f10825462297fa671b65491815
I20260928 22:56:12.077329 28169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:56:12.077692 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.066591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40312 (local address 127.14.14.190:44663)
0928 22:56:12.066786 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:12.066794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:12.066820 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:56:12.067224 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:12.067231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:12.067277 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:12.067457 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:12.068140 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.069483 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.070344 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.070980 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.071100 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.073758 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:12.073761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:12.073762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:12.073773 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:12.075901 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:12.076334 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:12.076338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:12.076338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:12.076442 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:12.076663 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:12.076664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:12.076664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:12.076989 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:12.077157 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:12.077309 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 22:56:12.077467 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260928 22:56:12.087642 28169 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:56:12.088160 28169 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:56:12.094933 28169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: Generated new TSK 0
I20260928 22:56:12.095125 28169 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:56:12.124281 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44663
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:12.208336 28181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:12.208670 28181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:12.208736 28181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:12.210354 28181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:12.210403 28181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:12.210433 28181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:56:12.211938 28181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:12.212196 28181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:12.212453 28181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:12.218051 28189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:12.218052 28186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:12.218101 28187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:12.218194 28181 server_base.cc:1034] running on GCE node
I20260928 22:56:12.218433 28181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:12.218683 28181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:12.219846 28181 hybrid_clock.cc:648] HybridClock initialized: now 1790636172219829 us; error 37 us; skew 500 ppm
Sep 28 22:56:12 dist-test-slave-kzfr krb5kdc[28074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:12.222453 28181 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:56:12.222903 28181 webserver.cc:460] Webserver started at http://127.14.14.129:34333/ using document root <none> and password file <none>
I20260928 22:56:12.223132 28181 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:12.223177 28181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:12.223348 28181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:56:12.224413 28181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "b9ba290a5c3841cbb20f14d27b800057"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "77181d70b3de5e05a975c035af9a3ecd"
server_key_iv: "4938a71f89fba0e3f01c2778ffbad8a7"
server_key_version: "encryptionkey@0"
I20260928 22:56:12.224773 28181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "b9ba290a5c3841cbb20f14d27b800057"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "77181d70b3de5e05a975c035af9a3ecd"
server_key_iv: "4938a71f89fba0e3f01c2778ffbad8a7"
server_key_version: "encryptionkey@0"
I20260928 22:56:12.226267 28181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260928 22:56:12.227119 28196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:12.227355 28181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:12.227470 28181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "b9ba290a5c3841cbb20f14d27b800057"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "77181d70b3de5e05a975c035af9a3ecd"
server_key_iv: "4938a71f89fba0e3f01c2778ffbad8a7"
server_key_version: "encryptionkey@0"
I20260928 22:56:12.227558 28181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:12.259377 28181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:12.261662 28181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:12.261844 28181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:12.262099 28181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:12.262420 28181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:12.262475 28181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:12.262549 28181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:12.262581 28181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:12.268397 28181 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:39707
I20260928 22:56:12.268455 28309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:39707 every 8 connection(s)
I20260928 22:56:12.268824 28181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:56:12.269879 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 28181
I20260928 22:56:12.270027 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:56:12.270234 14394 external_mini_cluster.cc:1428] Setting key 5d32375a99f4742f835fea1f85b014e7
Sep 28 22:56:12 dist-test-slave-kzfr krb5kdc[28074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
I20260928 22:56:12.280357 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.269601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:57487 (local address 127.14.14.190:44663)
0928 22:56:12.269671 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:12.269674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:12.269939 (+ 265us) server_negotiation.cc:407] Connection header received
0928 22:56:12.270618 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:12.270619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:12.270633 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:12.270679 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:12.271687 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.272446 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.273764 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.274113 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.274189 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.276940 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:12.276942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:12.276943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:12.276951 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:12.278783 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:12.279189 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:12.279190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:12.279191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:12.279266 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:12.279574 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:12.279575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:12.279575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:12.279686 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:12.279749 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:12.280189 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 22:56:12.280289 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:56:12.280562 28312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.269559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:57487)
0928 22:56:12.269811 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:12.269849 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:12.270498 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:12.270787 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:12.270796 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:12.271391 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:12.271581 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.271586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.272569 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.272570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.273659 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.273662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.274211 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.274212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.274406 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.275009 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:12.275018 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:12.276840 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:12.278885 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:12.278891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:12.278893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:12.279108 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:12.279353 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:12.279354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:12.279354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:12.279494 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:12.279780 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:12.279786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:12.279943 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 22:56:12.280047 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 22:56:12.280294 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":146,"threads_started":1}
I20260928 22:56:12.281278 28310 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44663
I20260928 22:56:12.281416 28310 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:12.281639 28310 heartbeater.cc:511] Master 127.14.14.190:44663 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
I20260928 22:56:12.282246 28119 ts_manager.cc:194] Registered new tserver with Master: b9ba290a5c3841cbb20f14d27b800057 (127.14.14.129:39707)
I20260928 22:56:12.282902 28119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:57487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:56:12.297153 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44663
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:12.377759 28315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:12.377977 28315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:12.378001 28315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:12.379098 28315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:12.379135 28315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:12.379161 28315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:56:12.380579 28315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:12.380765 28315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:12.380942 28315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:12.386169 28321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:12.386304 28323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:12.386188 28315 server_base.cc:1034] running on GCE node
W20260928 22:56:12.386215 28320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:12.386691 28315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:12.386996 28315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:12.388185 28315 hybrid_clock.cc:648] HybridClock initialized: now 1790636172388176 us; error 36 us; skew 500 ppm
Sep 28 22:56:12 dist-test-slave-kzfr krb5kdc[28074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:12.390837 28315 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:56:12.391294 28315 webserver.cc:460] Webserver started at http://127.14.14.130:44085/ using document root <none> and password file <none>
I20260928 22:56:12.391479 28315 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:12.391520 28315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:12.391659 28315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:56:12.392710 28315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "c3689bf232744d9cbec0b595e533deef"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "d4b22cbd1e5a709b52335794a0189d6b"
server_key_iv: "0f614319f386051befa22c43ec2991ac"
server_key_version: "encryptionkey@0"
I20260928 22:56:12.393066 28315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "c3689bf232744d9cbec0b595e533deef"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "d4b22cbd1e5a709b52335794a0189d6b"
server_key_iv: "0f614319f386051befa22c43ec2991ac"
server_key_version: "encryptionkey@0"
I20260928 22:56:12.394445 28315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:56:12.395208 28330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:12.395426 28315 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:56:12.395494 28315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "c3689bf232744d9cbec0b595e533deef"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "d4b22cbd1e5a709b52335794a0189d6b"
server_key_iv: "0f614319f386051befa22c43ec2991ac"
server_key_version: "encryptionkey@0"
I20260928 22:56:12.395583 28315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:12.412499 28315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:12.414650 28315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:12.414822 28315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:12.415058 28315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:12.415373 28315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:12.415405 28315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:12.415494 28315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:12.415517 28315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:12.421458 28315 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:43083
I20260928 22:56:12.421509 28443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:43083 every 8 connection(s)
I20260928 22:56:12.421844 28315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:56:12.422061 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 28315
I20260928 22:56:12.422181 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:56:12.422400 14394 external_mini_cluster.cc:1428] Setting key fe98069734705ab178197dbe8a32b741
Sep 28 22:56:12 dist-test-slave-kzfr krb5kdc[28074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
I20260928 22:56:12.432632 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.422662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:56659 (local address 127.14.14.190:44663)
0928 22:56:12.422737 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:12.422740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:12.422967 (+ 227us) server_negotiation.cc:407] Connection header received
0928 22:56:12.423712 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:12.423713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:12.423725 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:12.423779 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:12.425151 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.425996 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.427001 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.427301 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.427395 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.429552 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:12.429553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:12.429554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:12.429562 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:12.431149 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:12.431561 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:12.431562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:12.431562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:12.431618 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:12.431888 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:12.431889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:12.431889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:12.432032 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:12.432094 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:12.432465 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 22:56:12.432539 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 22:56:12.432806 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.422666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:56659)
0928 22:56:12.422824 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:12.422874 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:12.423603 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:12.423877 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:12.423881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:12.424759 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:12.425046 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.425051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.426101 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.426101 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.426908 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.426910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.427404 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.427405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.427540 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.428096 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:12.428104 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:12.429454 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:12.431249 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:12.431255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:12.431257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:12.431475 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:12.431696 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:12.431696 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:12.431697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:12.431812 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:12.432200 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:12.432205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:12.432325 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 22:56:12.432429 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 22:56:12.432580 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260928 22:56:12.433321 28444 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44663
I20260928 22:56:12.433423 28444 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
I20260928 22:56:12.433692 28444 heartbeater.cc:511] Master 127.14.14.190:44663 requested a full tablet report, sending...
I20260928 22:56:12.434141 28119 ts_manager.cc:194] Registered new tserver with Master: c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083)
I20260928 22:56:12.434525 28119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:56659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:56:12.448773 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44663
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:12.530643 28449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:12.530840 28449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:12.530864 28449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:12.532037 28449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:12.532078 28449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:12.532104 28449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:56:12.533556 28449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:12.533833 28449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:12.534075 28449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:12.539000 28457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:12.539016 28454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:12.539021 28455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:12.539279 28449 server_base.cc:1034] running on GCE node
I20260928 22:56:12.539499 28449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:12.539752 28449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:12.540911 28449 hybrid_clock.cc:648] HybridClock initialized: now 1790636172540906 us; error 44 us; skew 500 ppm
Sep 28 22:56:12 dist-test-slave-kzfr krb5kdc[28074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:12.543349 28449 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:56:12.543795 28449 webserver.cc:460] Webserver started at http://127.14.14.131:46117/ using document root <none> and password file <none>
I20260928 22:56:12.544028 28449 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:12.544095 28449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:12.544245 28449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:56:12.545269 28449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "072909ddd02b4be4b0fb286ddd99eeaf"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "08744b3a71e7a5f155b4864b6fdae481"
server_key_iv: "835821033214daff91467e2f4adaca58"
server_key_version: "encryptionkey@0"
I20260928 22:56:12.545681 28449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "072909ddd02b4be4b0fb286ddd99eeaf"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "08744b3a71e7a5f155b4864b6fdae481"
server_key_iv: "835821033214daff91467e2f4adaca58"
server_key_version: "encryptionkey@0"
I20260928 22:56:12.546993 28449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:56:12.547662 28464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:12.547817 28449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:56:12.547928 28449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "072909ddd02b4be4b0fb286ddd99eeaf"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "08744b3a71e7a5f155b4864b6fdae481"
server_key_iv: "835821033214daff91467e2f4adaca58"
server_key_version: "encryptionkey@0"
I20260928 22:56:12.548010 28449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:12.562336 28449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:12.564167 28449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:12.564323 28449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:12.564548 28449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:12.564888 28449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:12.564942 28449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:12.564983 28449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:12.565011 28449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:12.570919 28449 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:37617
I20260928 22:56:12.570955 28577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:37617 every 8 connection(s)
I20260928 22:56:12.571286 28449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:56:12.573760 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 28449
I20260928 22:56:12.573858 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:56:12.574087 14394 external_mini_cluster.cc:1428] Setting key 225e61105bcd8fdb7f9eac6145f0ceab
Sep 28 22:56:12 dist-test-slave-kzfr krb5kdc[28074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:12.580858 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.572059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:36853 (local address 127.14.14.190:44663)
0928 22:56:12.572137 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:12.572139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:12.572312 (+ 173us) server_negotiation.cc:407] Connection header received
0928 22:56:12.572800 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:12.572801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:12.572811 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:12.572863 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:12.574070 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.574947 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.576026 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.576210 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.576266 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.578529 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:12.578530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:12.578531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:12.578538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:12.579718 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:12.580066 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:12.580067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:12.580067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:12.580116 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:12.580372 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:12.580373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:12.580373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:12.580466 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:12.580515 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:12.580732 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 22:56:12.580790 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 22:56:12.581066 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.572014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:36853)
0928 22:56:12.572194 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:12.572230 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:12.572715 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:12.572915 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:12.572918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:12.573655 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:12.573967 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.573974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.575036 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.575038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.575949 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.575952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.576323 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.576324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.576424 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.576977 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:12.576985 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:12.578442 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:12.579789 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:12.579795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:12.579796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:12.579999 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:12.580172 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:12.580172 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:12.580173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:12.580308 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:12.580520 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:12.580524 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:12.580630 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 22:56:12.580709 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 22:56:12.580823 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260928 22:56:12.581559 28578 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44663
I20260928 22:56:12.581701 28578 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:12.581857 28578 heartbeater.cc:511] Master 127.14.14.190:44663 requested a full tablet report, sending...
I20260928 22:56:12.582185 28119 ts_manager.cc:194] Registered new tserver with Master: 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617)
I20260928 22:56:12.582576 28119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:36853
I20260928 22:56:12.585968 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:56:12.592953 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.587300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40328 (local address 127.14.14.190:44663)
0928 22:56:12.587362 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:12.587366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:12.587404 (+ 38us) server_negotiation.cc:407] Connection header received
0928 22:56:12.587466 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:12.587467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:12.587474 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:12.587513 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:12.588054 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.588560 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.589290 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.589465 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.589520 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.590720 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:12.590722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:12.590722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:12.590728 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:12.592090 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:12.592376 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:12.592376 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:12.592377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:12.592424 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:12.592571 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:12.592571 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:12.592571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:12.592690 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:12.592761 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:12.592839 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 22:56:12.592894 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:56:12.594421 28119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:56:12.596313 28119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:56:12.603039 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.599044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40722 (local address 127.14.14.129:39707)
0928 22:56:12.599202 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:12.599207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:12.599386 (+ 179us) server_negotiation.cc:407] Connection header received
0928 22:56:12.599432 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:12.599435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:12.599451 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:12.599524 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:12.599723 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.600864 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.602124 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.602597 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.602696 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.602753 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:12.602911 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 22:56:12.602954 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260928 22:56:12.603026 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.598812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:39707 (local address 127.0.0.1:40722)
0928 22:56:12.599254 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:12.599264 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:12.599332 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:12.599604 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:12.599605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:12.599607 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:12.599647 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.599648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.600969 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.600970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.602043 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.602045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.602699 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.602700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.602780 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.602794 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:56:12.602893 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 22:56:12.602940 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":32,"thread_start_us":97,"threads_started":1}
I20260928 22:56:12.603032 28589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.598783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37617 (local address 127.0.0.1:49828)
0928 22:56:12.599082 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:12.599092 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:12.599135 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:12.599385 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:12.599388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:12.599391 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:12.599504 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.599506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.600555 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.600556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.601792 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.601794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.602743 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.602744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.602808 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.602812 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:56:12.602870 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:56:12.602934 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":101,"threads_started":1}
I20260928 22:56:12.603191 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.599017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.14.14.131:37617)
0928 22:56:12.599202 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:12.599207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:12.599218 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:56:12.599260 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:12.599264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:12.599283 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:12.599584 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:12.599656 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.600441 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.601878 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.602585 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.602654 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.602709 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:12.603069 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 22:56:12.603116 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260928 22:56:12.603227 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.598781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:43083 (local address 127.0.0.1:36224)
0928 22:56:12.598962 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:12.599006 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:12.599110 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:12.599776 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:12.599777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:12.599779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:12.599827 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.599828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.601027 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.601028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.602086 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.602089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.603010 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.603011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.603076 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.603079 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:56:12.603127 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:56:12.603160 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260928 22:56:12.603461 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.598923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36224 (local address 127.14.14.130:43083)
0928 22:56:12.599480 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:12.599485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:12.599495 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:56:12.599597 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:12.599600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:12.599616 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:12.599694 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:12.599922 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.600947 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.602176 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.602873 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.602930 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.602977 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:12.603373 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 22:56:12.603403 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":76,"threads_started":1}
I20260928 22:56:12.603971 28512 tablet_service.cc:1467] Processing CreateTablet for tablet edd65499413e4728b817629eaca2d891 (DEFAULT_TABLE table=test-table [id=5120e8784acf4b05b156405b1220a785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:56:12.604117 28378 tablet_service.cc:1467] Processing CreateTablet for tablet edd65499413e4728b817629eaca2d891 (DEFAULT_TABLE table=test-table [id=5120e8784acf4b05b156405b1220a785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:56:12.604311 28512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd65499413e4728b817629eaca2d891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:12.604457 28378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd65499413e4728b817629eaca2d891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:12.604475 28243 tablet_service.cc:1467] Processing CreateTablet for tablet edd65499413e4728b817629eaca2d891 (DEFAULT_TABLE table=test-table [id=5120e8784acf4b05b156405b1220a785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:56:12.604748 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd65499413e4728b817629eaca2d891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:12.606937 28595 tablet_bootstrap.cc:492] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf: Bootstrap starting.
I20260928 22:56:12.607539 28595 tablet_bootstrap.cc:654] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:12.607926 28596 tablet_bootstrap.cc:492] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: Bootstrap starting.
I20260928 22:56:12.608318 28595 log.cc:824] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:12.608600 28596 tablet_bootstrap.cc:654] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:12.608750 28597 tablet_bootstrap.cc:492] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef: Bootstrap starting.
I20260928 22:56:12.609190 28596 log.cc:824] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:12.609427 28597 tablet_bootstrap.cc:654] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:12.610034 28597 log.cc:824] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:12.610061 28596 tablet_bootstrap.cc:492] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: No bootstrap required, opened a new log
I20260928 22:56:12.610124 28596 ts_tablet_manager.cc:1397] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:56:12.610576 28595 tablet_bootstrap.cc:492] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf: No bootstrap required, opened a new log
I20260928 22:56:12.610648 28595 ts_tablet_manager.cc:1397] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 22:56:12.612097 28596 raft_consensus.cc:348] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:12.612289 28596 raft_consensus.cc:374] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:12.612341 28596 raft_consensus.cc:729] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ba290a5c3841cbb20f14d27b800057, State: Initialized, Role: FOLLOWER
I20260928 22:56:12.612462 28596 consensus_queue.cc:260] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:12.612597 28595 raft_consensus.cc:348] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:12.612763 28596 ts_tablet_manager.cc:1428] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:56:12.612777 28595 raft_consensus.cc:374] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:12.612785 28310 heartbeater.cc:503] Master 127.14.14.190:44663 was elected leader, sending a full tablet report...
I20260928 22:56:12.612815 28595 raft_consensus.cc:729] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 072909ddd02b4be4b0fb286ddd99eeaf, State: Initialized, Role: FOLLOWER
I20260928 22:56:12.612929 28595 consensus_queue.cc:260] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:12.613193 28595 ts_tablet_manager.cc:1428] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:56:12.613199 28578 heartbeater.cc:503] Master 127.14.14.190:44663 was elected leader, sending a full tablet report...
I20260928 22:56:12.613663 28597 tablet_bootstrap.cc:492] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef: No bootstrap required, opened a new log
I20260928 22:56:12.613732 28597 ts_tablet_manager.cc:1397] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 22:56:12.615311 28597 raft_consensus.cc:348] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:12.615453 28597 raft_consensus.cc:374] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:12.615496 28597 raft_consensus.cc:729] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3689bf232744d9cbec0b595e533deef, State: Initialized, Role: FOLLOWER
I20260928 22:56:12.615607 28597 consensus_queue.cc:260] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:12.615873 28597 ts_tablet_manager.cc:1428] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:56:12.615926 28444 heartbeater.cc:503] Master 127.14.14.190:44663 was elected leader, sending a full tablet report...
W20260928 22:56:12.672713 28445 tablet.cc:2369] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:56:12.769630 28311 tablet.cc:2369] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:56:12.821799 28579 tablet.cc:2369] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:56:12.887421 28601 raft_consensus.cc:482] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:56:12.887554 28601 raft_consensus.cc:504] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:12.887884 28601 leader_election.cc:288] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617), c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083)
I20260928 22:56:12.892463 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.888272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:53209 (local address 127.14.14.131:37617)
0928 22:56:12.888322 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:12.888325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:12.888376 (+ 51us) server_negotiation.cc:407] Connection header received
0928 22:56:12.888441 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:12.888442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:12.888451 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:12.888529 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:12.888752 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.889575 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.891481 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.892073 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.892155 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.892164 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:12.892341 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 22:56:12.892376 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:56:12.892494 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.888096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37617 (local address 127.14.14.129:53209)
0928 22:56:12.888287 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:12.888300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:12.888378 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:12.888554 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:12.888555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:12.888559 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:12.888657 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.888660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.889742 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.889743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.891373 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.891377 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.892171 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.892172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.892256 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.892261 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:56:12.892327 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 22:56:12.892382 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":126,"thread_start_us":91,"threads_started":1}
I20260928 22:56:12.892530 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.888094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:43083 (local address 127.14.14.129:34739)
0928 22:56:12.888287 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:12.888300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:12.888363 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:12.888554 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:12.888555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:12.888558 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:12.888664 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.888665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.889741 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.889743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.891374 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:12.891377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.892173 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.892174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:12.892351 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.892354 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:56:12.892411 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:56:12.892444 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":38,"spinlock_wait_cycles":14080,"thread_start_us":96,"threads_started":1}
I20260928 22:56:12.892535 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.888198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:34739 (local address 127.14.14.130:43083)
0928 22:56:12.888342 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:12.888344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:12.888355 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:56:12.888437 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:12.888439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:12.888451 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:12.888529 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:12.888752 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.889655 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.891480 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.892068 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.892154 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.892164 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:12.892428 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 22:56:12.892467 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260928 22:56:12.892976 28398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd65499413e4728b817629eaca2d891" candidate_uuid: "b9ba290a5c3841cbb20f14d27b800057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3689bf232744d9cbec0b595e533deef" is_pre_election: true
I20260928 22:56:12.893039 28532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd65499413e4728b817629eaca2d891" candidate_uuid: "b9ba290a5c3841cbb20f14d27b800057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" is_pre_election: true
I20260928 22:56:12.893177 28532 raft_consensus.cc:2454] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9ba290a5c3841cbb20f14d27b800057 in term 0.
I20260928 22:56:12.893172 28398 raft_consensus.cc:2454] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9ba290a5c3841cbb20f14d27b800057 in term 0.
I20260928 22:56:12.893427 28199 leader_election.cc:302] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 072909ddd02b4be4b0fb286ddd99eeaf, b9ba290a5c3841cbb20f14d27b800057; no voters:
I20260928 22:56:12.893608 28601 raft_consensus.cc:2781] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:56:12.893672 28601 raft_consensus.cc:482] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:56:12.893709 28601 raft_consensus.cc:3037] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:12.894440 28601 raft_consensus.cc:504] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:12.894603 28601 leader_election.cc:288] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 1 election: Requested vote from peers 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617), c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083)
I20260928 22:56:12.894779 28532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd65499413e4728b817629eaca2d891" candidate_uuid: "b9ba290a5c3841cbb20f14d27b800057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "072909ddd02b4be4b0fb286ddd99eeaf"
I20260928 22:56:12.894811 28398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd65499413e4728b817629eaca2d891" candidate_uuid: "b9ba290a5c3841cbb20f14d27b800057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3689bf232744d9cbec0b595e533deef"
I20260928 22:56:12.894867 28532 raft_consensus.cc:3037] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:12.894917 28398 raft_consensus.cc:3037] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:12.895651 28532 raft_consensus.cc:2454] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9ba290a5c3841cbb20f14d27b800057 in term 1.
I20260928 22:56:12.895735 28398 raft_consensus.cc:2454] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9ba290a5c3841cbb20f14d27b800057 in term 1.
I20260928 22:56:12.895845 28199 leader_election.cc:302] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 072909ddd02b4be4b0fb286ddd99eeaf, b9ba290a5c3841cbb20f14d27b800057; no voters:
I20260928 22:56:12.895963 28601 raft_consensus.cc:2781] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:56:12.896140 28601 raft_consensus.cc:686] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [term 1 LEADER]: Becoming Leader. State: Replica: b9ba290a5c3841cbb20f14d27b800057, State: Running, Role: LEADER
I20260928 22:56:12.896240 28601 consensus_queue.cc:237] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:12.897081 28119 catalog_manager.cc:5495] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9ba290a5c3841cbb20f14d27b800057 (127.14.14.129). New cstate: current_term: 1 leader_uuid: "b9ba290a5c3841cbb20f14d27b800057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } health_report { overall_health: HEALTHY } } }
I20260928 22:56:12.957798 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.954938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.14.14.129:39707)
0928 22:56:12.954996 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:12.954997 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:12.955006 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:56:12.955071 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:12.955072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:12.955091 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:12.955148 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:56:12.955409 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.955938 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.956923 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:12.957111 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:12.957170 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:12.957353 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:56:12.957460 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:56:12.957522 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:12.957719 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 22:56:12.957737 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:56:12.959825 28532 raft_consensus.cc:1261] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf [term 1 FOLLOWER]: Refusing update from remote peer b9ba290a5c3841cbb20f14d27b800057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:56:12.959826 28398 raft_consensus.cc:1261] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef [term 1 FOLLOWER]: Refusing update from remote peer b9ba290a5c3841cbb20f14d27b800057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:56:12.960198 28601 consensus_queue.cc:1035] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:56:12.960368 28606 consensus_queue.cc:1035] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:56:12.962713 28609 mvcc.cc:205] Tried to move back new op lower bound from 7334445764440584192 to 7334445764183535616. Current Snapshot: MvccSnapshot[applied={T|T < 7334445764440584192}]
I20260928 22:56:12.963379 28611 mvcc.cc:205] Tried to move back new op lower bound from 7334445764440584192 to 7334445764183535616. Current Snapshot: MvccSnapshot[applied={T|T < 7334445764440584192}]
I20260928 22:56:12.963624 28610 mvcc.cc:205] Tried to move back new op lower bound from 7334445764440584192 to 7334445764183535616. Current Snapshot: MvccSnapshot[applied={T|T < 7334445764440584192}]
I20260928 22:56:12.965328 28118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:56:12.965440 28118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:56:12.966254 28118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: Sending DeleteTablet for 3 replicas of tablet edd65499413e4728b817629eaca2d891
I20260928 22:56:12.966585 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet edd65499413e4728b817629eaca2d891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:12 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:40722
I20260928 22:56:12.966650 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet edd65499413e4728b817629eaca2d891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:12 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:36224
I20260928 22:56:12.966634 28512 tablet_service.cc:1514] Processing DeleteTablet for tablet edd65499413e4728b817629eaca2d891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:12 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:49828
I20260928 22:56:12.966848 28620 tablet_replica.cc:330] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef: stopping tablet replica
I20260928 22:56:12.966905 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 28181
I20260928 22:56:12.966945 28620 raft_consensus.cc:2229] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:56:12.967073 28620 raft_consensus.cc:2258] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:56:12.967658 28621 tablet_replica.cc:330] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf: stopping tablet replica
I20260928 22:56:12.967744 28620 ts_tablet_manager.cc:1905] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:56:12.967790 28621 raft_consensus.cc:2229] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:56:12.967914 28621 raft_consensus.cc:2258] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:56:12.969182 28621 ts_tablet_manager.cc:1905] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:56:12.969897 28620 ts_tablet_manager.cc:1918] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:56:12.969985 28620 log.cc:1196] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/edd65499413e4728b817629eaca2d891
I20260928 22:56:12.970247 28620 ts_tablet_manager.cc:1939] T edd65499413e4728b817629eaca2d891 P c3689bf232744d9cbec0b595e533deef: Deleting consensus metadata
I20260928 22:56:12.970705 28106 catalog_manager.cc:4841] TS c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083): tablet edd65499413e4728b817629eaca2d891 (table test-table [id=5120e8784acf4b05b156405b1220a785]) successfully deleted
I20260928 22:56:12.970857 28621 ts_tablet_manager.cc:1918] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:56:12.970929 28621 log.cc:1196] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/edd65499413e4728b817629eaca2d891
I20260928 22:56:12.971165 28621 ts_tablet_manager.cc:1939] T edd65499413e4728b817629eaca2d891 P 072909ddd02b4be4b0fb286ddd99eeaf: Deleting consensus metadata
I20260928 22:56:12.971763 28105 catalog_manager.cc:4841] TS 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617): tablet edd65499413e4728b817629eaca2d891 (table test-table [id=5120e8784acf4b05b156405b1220a785]) successfully deleted
W20260928 22:56:12.974697 28104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:39707 (error 108)
I20260928 22:56:12.975013 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 28315
I20260928 22:56:12.975651 28589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:12.975125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:39707 (local address 127.0.0.1:40754)
0928 22:56:12.975568 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:12.975608 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:39707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423}
I20260928 22:56:12.980288 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 28449
W20260928 22:56:12.980304 28104 catalog_manager.cc:4584] TS b9ba290a5c3841cbb20f14d27b800057 (127.14.14.129:39707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet edd65499413e4728b817629eaca2d891: Network error: Client connection negotiation failed: client connection to 127.14.14.129:39707: connect: Connection refused (error 111)
I20260928 22:56:12.984915 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 28087
I20260928 22:56:12.990607 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44663
--webserver_interface=127.14.14.190
--webserver_port=44509
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:44663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:13.073812 28623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:13.074061 28623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:13.074100 28623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:13.075297 28623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:56:13.075374 28623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:13.075402 28623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:13.075628 28623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:56:13.075721 28623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:56:13.077450 28623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:44663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=44509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:13.077771 28623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:13.078009 28623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:13.087631 28629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:13.087682 28632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:13.087848 28623 server_base.cc:1034] running on GCE node
W20260928 22:56:13.087631 28630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:13.088254 28623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:13.088569 28623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:13.089740 28623 hybrid_clock.cc:648] HybridClock initialized: now 1790636173089727 us; error 34 us; skew 500 ppm
Sep 28 22:56:13 dist-test-slave-kzfr krb5kdc[28074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:13.092653 28623 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:56:13.093145 28623 webserver.cc:460] Webserver started at http://127.14.14.190:44509/ using document root <none> and password file <none>
I20260928 22:56:13.093402 28623 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:13.093463 28623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:13.094956 28623 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:13.095688 28639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:13.095892 28623 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:13.095980 28623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "d89d2420acf6402cbba1f9fa04f70fa0"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "a1191e486fd9c8c723cc139fb1d9090e"
server_key_iv: "299d5b1ef61c70873ba3e2b90ebfaec2"
server_key_version: "encryptionkey@0"
I20260928 22:56:13.096374 28623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:13.113897 28623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:13.116382 28623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:13.116499 28623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:13.120345 28623 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:44663
I20260928 22:56:13.120420 28691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:44663 every 8 connection(s)
I20260928 22:56:13.120801 28623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:56:13.122530 28692 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:56:13.123226 28692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: Bootstrap starting.
I20260928 22:56:13.125792 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 28623
I20260928 22:56:13.126057 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:39707
--local_ip_for_outbound_sockets=127.14.14.129
--tserver_master_addrs=127.14.14.190:44663
--webserver_port=34333
--webserver_interface=127.14.14.129
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:56:13.131981 28692 log.cc:824] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:13.134080 28692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:56:13.134408 28692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: Bootstrap complete.
I20260928 22:56:13.136679 28692 raft_consensus.cc:348] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } }
I20260928 22:56:13.136962 28692 raft_consensus.cc:729] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d89d2420acf6402cbba1f9fa04f70fa0, State: Initialized, Role: FOLLOWER
I20260928 22:56:13.137151 28692 consensus_queue.cc:260] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } }
I20260928 22:56:13.137229 28692 raft_consensus.cc:388] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:56:13.137288 28692 raft_consensus.cc:482] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:56:13.137365 28692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:56:13.138468 28692 raft_consensus.cc:504] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } }
I20260928 22:56:13.138590 28692 leader_election.cc:302] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d89d2420acf6402cbba1f9fa04f70fa0; no voters:
I20260928 22:56:13.138796 28692 leader_election.cc:288] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 22:56:13.138832 28696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:56:13.138954 28696 raft_consensus.cc:686] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 2 LEADER]: Becoming Leader. State: Replica: d89d2420acf6402cbba1f9fa04f70fa0, State: Running, Role: LEADER
I20260928 22:56:13.139076 28692 sys_catalog.cc:564] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:56:13.139127 28696 consensus_queue.cc:237] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } }
I20260928 22:56:13.139488 28697 sys_catalog.cc:455] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } } }
I20260928 22:56:13.139487 28696 sys_catalog.cc:455] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d89d2420acf6402cbba1f9fa04f70fa0. Latest consensus state: current_term: 2 leader_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } } }
I20260928 22:56:13.139647 28697 sys_catalog.cc:458] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:13.139709 28696 sys_catalog.cc:458] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:13.140377 28705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:56:13.141415 28705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:56:13.141523 28705 catalog_manager.cc:1259] Loaded cluster ID: 84e1c2f10825462297fa671b65491815
I20260928 22:56:13.141557 28705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:56:13.141844 28705 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 22:56:13.147580 28705 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 22:56:13.150506 28705 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:56:13.150748 28705 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: Loaded TSK: 0
I20260928 22:56:13.150980 28705 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:56:13.218940 28694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:13.219163 28694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:13.219188 28694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:13.220607 28694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:13.220664 28694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:13.220692 28694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:56:13.222306 28694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:39707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=34333
--tserver_master_addrs=127.14.14.190:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:13.222564 28694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:13.222837 28694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:13.228122 28722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:13.228127 28719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:13.228305 28694 server_base.cc:1034] running on GCE node
W20260928 22:56:13.228488 28720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:13.228739 28694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:13.229018 28694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:13.230204 28694 hybrid_clock.cc:648] HybridClock initialized: now 1790636173230172 us; error 46 us; skew 500 ppm
Sep 28 22:56:13 dist-test-slave-kzfr krb5kdc[28074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:13.232898 28694 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:56:13.233389 28694 webserver.cc:460] Webserver started at http://127.14.14.129:34333/ using document root <none> and password file <none>
I20260928 22:56:13.233698 28694 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:13.233794 28694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:13.235210 28694 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:13.235957 28729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:13.236143 28694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:13.236245 28694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "b9ba290a5c3841cbb20f14d27b800057"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "77181d70b3de5e05a975c035af9a3ecd"
server_key_iv: "4938a71f89fba0e3f01c2778ffbad8a7"
server_key_version: "encryptionkey@0"
I20260928 22:56:13.236601 28694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:13.252209 28694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:13.254667 28694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:13.254854 28694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:13.255118 28694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:13.255666 28736 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:56:13.257548 28694 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:56:13.257694 28694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 22:56:13.257750 28694 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:56:13.258430 28694 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:56:13.258491 28694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 22:56:13.258549 28736 tablet_bootstrap.cc:492] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: Bootstrap starting.
I20260928 22:56:13.265885 28694 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:39707
I20260928 22:56:13.265928 28843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:39707 every 8 connection(s)
I20260928 22:56:13.266320 28694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:56:13.269500 28736 log.cc:824] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:13.272064 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 28694
I20260928 22:56:13.272334 28736 tablet_bootstrap.cc:492] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:56:13.272372 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:43083
--local_ip_for_outbound_sockets=127.14.14.130
--tserver_master_addrs=127.14.14.190:44663
--webserver_port=44085
--webserver_interface=127.14.14.130
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:56:13.272644 28736 tablet_bootstrap.cc:492] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: Bootstrap complete.
I20260928 22:56:13.273931 28736 ts_tablet_manager.cc:1397] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.012s
I20260928 22:56:13.275329 28736 raft_consensus.cc:348] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:13.275647 28736 raft_consensus.cc:729] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ba290a5c3841cbb20f14d27b800057, State: Initialized, Role: FOLLOWER
I20260928 22:56:13.275813 28736 consensus_queue.cc:260] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:13.276054 28736 ts_tablet_manager.cc:1428] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Sep 28 22:56:13 dist-test-slave-kzfr krb5kdc[28074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:13.282646 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.267127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:36917 (local address 127.14.14.190:44663)
0928 22:56:13.270203 (+ 3076us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:13.270212 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:13.270271 (+ 59us) server_negotiation.cc:407] Connection header received
0928 22:56:13.270735 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:13.270741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:13.270789 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:13.270934 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:13.271619 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.273399 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.274769 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.275416 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.275533 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.278644 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:13.278650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:13.278651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:13.278673 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:13.280475 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:13.280949 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:13.280955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:13.280956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:13.281058 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:13.281511 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:13.281513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:13.281514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:13.281857 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:13.281964 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:13.282294 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 22:56:13.282438 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3018,"thread_start_us":79,"threads_started":1}
I20260928 22:56:13.282660 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.267048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:36917)
0928 22:56:13.269782 (+ 2734us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:13.269839 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:13.270497 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:13.270964 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:13.270968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:13.271287 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:13.271494 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.271499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.273520 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.273521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.274644 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.274647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.275632 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.275633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.276197 (+ 564us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.276890 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:13.276898 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:13.278517 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:13.280594 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:13.280602 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:13.280604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:13.280848 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:13.281137 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:13.281138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:13.281139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:13.281388 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:13.281942 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:13.281949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:13.282096 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 22:56:13.282210 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 22:56:13.282438 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2643,"thread_start_us":93,"threads_started":1}
I20260928 22:56:13.283933 28844 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44663
I20260928 22:56:13.284102 28844 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:13.284300 28844 heartbeater.cc:511] Master 127.14.14.190:44663 requested a full tablet report, sending...
I20260928 22:56:13.285032 28656 ts_manager.cc:194] Registered new tserver with Master: b9ba290a5c3841cbb20f14d27b800057 (127.14.14.129:39707)
I20260928 22:56:13.286284 28656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:36917
I20260928 22:56:13.287334 28844 heartbeater.cc:503] Master 127.14.14.190:44663 was elected leader, sending a full tablet report...
Sep 28 22:56:13 dist-test-slave-kzfr krb5kdc[28074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for kudu/127.14.14.129@KRBTEST.COM
I20260928 22:56:13.295390 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.285912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:39707 (local address 127.0.0.1:48586)
0928 22:56:13.286155 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:13.286197 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:13.286276 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:13.286860 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:13.286866 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:13.287190 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:13.287260 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.287263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.288618 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.288619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.289184 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.289186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.289890 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.289891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.289955 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.290487 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:13.290491 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:13.292045 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:13.294151 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:13.294158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:13.294160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:13.294367 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:13.294616 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:13.294617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:13.294617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:13.294684 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:13.295000 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:13.295006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:13.295059 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 22:56:13.295159 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 22:56:13.295287 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":64,"threads_started":1}
I20260928 22:56:13.295411 28851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.285723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.14.14.129:39707)
0928 22:56:13.286101 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:13.286105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:13.286445 (+ 340us) server_negotiation.cc:407] Connection header received
0928 22:56:13.286484 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:13.286486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:13.286501 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:13.286784 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:13.287372 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.288524 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.289298 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.289802 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.289874 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.292131 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:13.292139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:13.292140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:13.292147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:13.294044 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:13.294476 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:13.294479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:13.294480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:13.294542 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:13.294772 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:13.294773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:13.294773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:13.294918 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:13.295010 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:13.295225 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 22:56:13.295315 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":97,"threads_started":1}
I20260928 22:56:13.295815 28778 tablet_service.cc:1514] Processing DeleteTablet for tablet edd65499413e4728b817629eaca2d891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:12 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:48586
I20260928 22:56:13.296417 28853 tablet_replica.cc:330] stopping tablet replica
I20260928 22:56:13.296533 28853 raft_consensus.cc:2229] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:56:13.296600 28853 raft_consensus.cc:2258] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:56:13.296911 28853 ts_tablet_manager.cc:1905] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:56:13.299592 28853 ts_tablet_manager.cc:1918] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:56:13.299672 28853 log.cc:1196] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal/wals/edd65499413e4728b817629eaca2d891
I20260928 22:56:13.299908 28853 ts_tablet_manager.cc:1939] T edd65499413e4728b817629eaca2d891 P b9ba290a5c3841cbb20f14d27b800057: Deleting consensus metadata
I20260928 22:56:13.300437 28641 catalog_manager.cc:4841] TS b9ba290a5c3841cbb20f14d27b800057 (127.14.14.129:39707): tablet edd65499413e4728b817629eaca2d891 (table test-table [id=5120e8784acf4b05b156405b1220a785]) successfully deleted
W20260928 22:56:13.362166 28849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:13.362365 28849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:13.362388 28849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:13.363556 28849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:13.363600 28849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:13.363626 28849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:56:13.365114 28849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:43083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=44085
--tserver_master_addrs=127.14.14.190:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:13.365352 28849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:13.365576 28849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:13.370905 28858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:13.370913 28859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:13.371029 28849 server_base.cc:1034] running on GCE node
W20260928 22:56:13.370924 28861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:13.371415 28849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:13.371678 28849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:13.372880 28849 hybrid_clock.cc:648] HybridClock initialized: now 1790636173372864 us; error 29 us; skew 500 ppm
Sep 28 22:56:13 dist-test-slave-kzfr krb5kdc[28074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:13.375573 28849 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:56:13.376025 28849 webserver.cc:460] Webserver started at http://127.14.14.130:44085/ using document root <none> and password file <none>
I20260928 22:56:13.376283 28849 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:13.376381 28849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:13.377878 28849 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:56:13.378582 28868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:13.378768 28849 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:56:13.378866 28849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "c3689bf232744d9cbec0b595e533deef"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "d4b22cbd1e5a709b52335794a0189d6b"
server_key_iv: "0f614319f386051befa22c43ec2991ac"
server_key_version: "encryptionkey@0"
I20260928 22:56:13.379311 28849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:13.398362 28849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:13.400940 28849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:13.401124 28849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:13.401404 28849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:13.401752 28849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:13.401813 28849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:13.401854 28849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:13.401885 28849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:13.408418 28849 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:43083
I20260928 22:56:13.408468 28981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:43083 every 8 connection(s)
I20260928 22:56:13.408833 28849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:56:13.410061 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 28849
I20260928 22:56:13.410357 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:37617
--local_ip_for_outbound_sockets=127.14.14.131
--tserver_master_addrs=127.14.14.190:44663
--webserver_port=46117
--webserver_interface=127.14.14.131
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:56:13 dist-test-slave-kzfr krb5kdc[28074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:13.420579 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.409625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:58379 (local address 127.14.14.190:44663)
0928 22:56:13.409741 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:13.409744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:13.409921 (+ 177us) server_negotiation.cc:407] Connection header received
0928 22:56:13.410694 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:13.410695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:13.410706 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:13.410762 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:13.412115 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.412622 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.414238 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.414470 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.414532 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.417603 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:13.417604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:13.417605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:13.417613 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:13.419126 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:13.419569 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:13.419571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:13.419571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:13.419634 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:13.419904 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:13.419905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:13.419905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:13.420023 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:13.420100 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:13.420396 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 22:56:13.420486 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 22:56:13.420769 28984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.409564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:58379)
0928 22:56:13.409745 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:13.409800 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:13.410592 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:13.410839 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:13.410843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:13.411679 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:13.411984 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.411990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.412733 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.412734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.414138 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.414142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.414570 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.414571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.414731 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.415694 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:13.415706 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:13.417471 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:13.419227 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:13.419234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:13.419236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:13.419469 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:13.419707 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:13.419708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:13.419708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:13.419829 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:13.420103 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:13.420110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:13.420258 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 22:56:13.420374 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 22:56:13.420533 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260928 22:56:13.421408 28982 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44663
I20260928 22:56:13.421543 28982 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:13.421748 28982 heartbeater.cc:511] Master 127.14.14.190:44663 requested a full tablet report, sending...
I20260928 22:56:13.422196 28656 ts_manager.cc:194] Registered new tserver with Master: c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083)
I20260928 22:56:13.422700 28656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:58379
W20260928 22:56:13.495386 28985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:13.495620 28985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:13.495645 28985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:13.496752 28985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:13.496785 28985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:13.496811 28985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:56:13.498257 28985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:37617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=46117
--tserver_master_addrs=127.14.14.190:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:13.498453 28985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:13.498704 28985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:13.504088 28990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:13.504099 28985 server_base.cc:1034] running on GCE node
W20260928 22:56:13.504082 28993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:13.504061 28991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:13.504532 28985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:13.504793 28985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:13.505936 28985 hybrid_clock.cc:648] HybridClock initialized: now 1790636173505925 us; error 28 us; skew 500 ppm
Sep 28 22:56:13 dist-test-slave-kzfr krb5kdc[28074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:13.508437 28985 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:56:13.508829 28985 webserver.cc:460] Webserver started at http://127.14.14.131:46117/ using document root <none> and password file <none>
I20260928 22:56:13.509024 28985 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:13.509074 28985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:13.510437 28985 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:13.510984 29000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:13.511204 28985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:13.511303 28985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "072909ddd02b4be4b0fb286ddd99eeaf"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "08744b3a71e7a5f155b4864b6fdae481"
server_key_iv: "835821033214daff91467e2f4adaca58"
server_key_version: "encryptionkey@0"
I20260928 22:56:13.511658 28985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:13.519395 28985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:13.521296 28985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:13.521476 28985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:13.521737 28985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:13.522043 28985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:13.522099 28985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:13.522145 28985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:13.522176 28985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:13.528028 28985 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:37617
I20260928 22:56:13.528080 29113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:37617 every 8 connection(s)
I20260928 22:56:13.528462 28985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:56:13 dist-test-slave-kzfr krb5kdc[28074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:13.536144 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 28985
I20260928 22:56:13.538287 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.529280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:55415 (local address 127.14.14.190:44663)
0928 22:56:13.529353 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:13.529355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:13.529575 (+ 220us) server_negotiation.cc:407] Connection header received
0928 22:56:13.530147 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:13.530149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:13.530161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:13.530216 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:13.531161 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.531665 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.532691 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.532917 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.533010 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.535175 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:13.535177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:13.535177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:13.535185 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:13.536645 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:13.537195 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:13.537197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:13.537197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:13.537273 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:13.537551 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:13.537552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:13.537552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:13.537712 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:13.537775 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:13.538139 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 22:56:13.538223 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:56:13.538475 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.529264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:55415)
0928 22:56:13.529455 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:13.529490 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:13.530033 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:13.530303 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:13.530306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:13.530863 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:13.531054 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.531058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.531766 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.531767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.532590 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.532592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.533020 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.533021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.533156 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.533689 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:13.533696 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:13.535068 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:13.536724 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:13.536731 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:13.536733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:13.537102 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:13.537353 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:13.537354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:13.537354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:13.537482 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:13.537802 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:13.537809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:13.537964 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 22:56:13.538083 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 22:56:13.538256 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
I20260928 22:56:13.538960 29114 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44663
I20260928 22:56:13.539057 29114 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:13.539283 29114 heartbeater.cc:511] Master 127.14.14.190:44663 requested a full tablet report, sending...
I20260928 22:56:13.539673 28656 ts_manager.cc:194] Registered new tserver with Master: 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617)
I20260928 22:56:13.540136 28656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:55415
I20260928 22:56:13.544019 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.536592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.14.14.190:44663)
0928 22:56:13.536698 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:13.536700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:13.536719 (+ 19us) server_negotiation.cc:407] Connection header received
0928 22:56:13.536807 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:13.536807 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:13.536820 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:13.536894 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:13.537772 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.538633 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.539711 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.539991 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.540073 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.541341 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:13.541342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:13.541343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:13.541349 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:13.542813 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:13.543196 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:13.543197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:13.543197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:13.543256 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:13.543479 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:13.543480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:13.543480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:13.543595 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:13.543787 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:13.543824 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 22:56:13.543916 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":62,"threads_started":1}
I20260928 22:56:13.544730 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:56:13.547709 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.545117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54050 (local address 127.14.14.190:44663)
0928 22:56:13.545184 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:13.545266 (+ 82us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:13.545276 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:56:13.545355 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:13.545356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:13.545392 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:13.545483 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:56:13.545644 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.546112 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.546889 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.547113 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.547203 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.547367 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:56:13.547478 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:56:13.547590 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:13.547608 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 22:56:13.547629 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:56:13.548018 28656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:56:13.549715 28656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:56:13.552389 28778 tablet_service.cc:1467] Processing CreateTablet for tablet 29534103ecf5486d9ff523c1855ae8af (DEFAULT_TABLE table=test-table [id=3f9be8d67ead4cdcb11d72c1e16c51e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:56:13.552630 28778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29534103ecf5486d9ff523c1855ae8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:13.554531 28736 tablet_bootstrap.cc:492] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057: Bootstrap starting.
I20260928 22:56:13.555114 28736 tablet_bootstrap.cc:654] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:13.555931 28736 tablet_bootstrap.cc:492] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057: No bootstrap required, opened a new log
I20260928 22:56:13.555995 28736 ts_tablet_manager.cc:1397] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:56:13.556378 28736 raft_consensus.cc:348] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:13.556466 28736 raft_consensus.cc:374] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:13.556541 28736 raft_consensus.cc:729] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ba290a5c3841cbb20f14d27b800057, State: Initialized, Role: FOLLOWER
I20260928 22:56:13.556605 28736 consensus_queue.cc:260] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:13.556716 28736 ts_tablet_manager.cc:1428] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:56:13.557126 29120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.552087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:43083 (local address 127.0.0.1:39972)
0928 22:56:13.552402 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:13.552411 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:13.552458 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:13.552773 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:13.552774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:13.552777 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:13.552831 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.552833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.554473 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.554474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.555791 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.555795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.556832 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.556833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.556891 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.556939 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 22:56:13.556997 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:56:13.557043 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":152,"thread_start_us":97,"threads_started":1}
I20260928 22:56:13.557433 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.552316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39972 (local address 127.14.14.130:43083)
0928 22:56:13.552506 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:13.552512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:13.552523 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:56:13.552568 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:13.552570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:13.552589 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:13.552698 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:13.552930 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.554345 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.555886 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.556659 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.556731 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.556797 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:13.557292 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 22:56:13.557348 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260928 22:56:13.558238 28916 tablet_service.cc:1467] Processing CreateTablet for tablet 29534103ecf5486d9ff523c1855ae8af (DEFAULT_TABLE table=test-table [id=3f9be8d67ead4cdcb11d72c1e16c51e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:56:13.558645 28916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29534103ecf5486d9ff523c1855ae8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:13.559136 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.552087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37617 (local address 127.0.0.1:45826)
0928 22:56:13.552142 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:13.552153 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:13.552193 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:13.555068 (+ 2875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:13.555069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:13.555072 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:13.555122 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.555124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.556602 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.556603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.557937 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.557940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.558934 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.558935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.558996 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.559000 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:56:13.559054 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 22:56:13.559090 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 22:56:13.559201 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.552232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45826 (local address 127.14.14.131:37617)
0928 22:56:13.554837 (+ 2605us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:13.554844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:13.554857 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:56:13.554916 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:13.554918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:13.554937 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:13.555215 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:13.555319 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.556460 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.558047 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.558849 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.558931 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.558990 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:13.559069 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 22:56:13.559112 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2524,"thread_start_us":105,"threads_started":1}
I20260928 22:56:13.559809 29048 tablet_service.cc:1467] Processing CreateTablet for tablet 29534103ecf5486d9ff523c1855ae8af (DEFAULT_TABLE table=test-table [id=3f9be8d67ead4cdcb11d72c1e16c51e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:56:13.560103 29048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29534103ecf5486d9ff523c1855ae8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:13.562132 29124 tablet_bootstrap.cc:492] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef: Bootstrap starting.
I20260928 22:56:13.562808 29124 tablet_bootstrap.cc:654] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:13.563364 29124 log.cc:824] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:13.563349 29125 tablet_bootstrap.cc:492] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf: Bootstrap starting.
I20260928 22:56:13.564214 29125 tablet_bootstrap.cc:654] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:13.564252 29124 tablet_bootstrap.cc:492] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef: No bootstrap required, opened a new log
I20260928 22:56:13.564324 29124 ts_tablet_manager.cc:1397] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:56:13.564757 29125 log.cc:824] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:13.565517 29125 tablet_bootstrap.cc:492] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf: No bootstrap required, opened a new log
I20260928 22:56:13.565631 29125 ts_tablet_manager.cc:1397] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:56:13.565824 29124 raft_consensus.cc:348] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:13.565956 29124 raft_consensus.cc:374] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:13.565977 29124 raft_consensus.cc:729] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3689bf232744d9cbec0b595e533deef, State: Initialized, Role: FOLLOWER
I20260928 22:56:13.566102 29124 consensus_queue.cc:260] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:13.566365 29124 ts_tablet_manager.cc:1428] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:56:13.566416 28982 heartbeater.cc:503] Master 127.14.14.190:44663 was elected leader, sending a full tablet report...
I20260928 22:56:13.567764 29125 raft_consensus.cc:348] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:13.567920 29125 raft_consensus.cc:374] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:13.567945 29125 raft_consensus.cc:729] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 072909ddd02b4be4b0fb286ddd99eeaf, State: Initialized, Role: FOLLOWER
I20260928 22:56:13.568089 29125 consensus_queue.cc:260] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:13.568409 29125 ts_tablet_manager.cc:1428] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:56:13.568444 29114 heartbeater.cc:503] Master 127.14.14.190:44663 was elected leader, sending a full tablet report...
I20260928 22:56:13.617630 29128 raft_consensus.cc:482] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:56:13.617762 29128 raft_consensus.cc:504] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:13.618131 29128 leader_election.cc:288] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617), b9ba290a5c3841cbb20f14d27b800057 (127.14.14.129:39707)
I20260928 22:56:13.622116 28851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.618490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:51221 (local address 127.14.14.129:39707)
0928 22:56:13.618533 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:13.618535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:13.618672 (+ 137us) server_negotiation.cc:407] Connection header received
0928 22:56:13.618711 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:13.618712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:13.618733 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:13.618822 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:13.618984 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.619707 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.621060 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.621744 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.621822 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.621903 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:13.622008 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 22:56:13.622048 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:56:13.622116 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.618390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:55913 (local address 127.14.14.131:37617)
0928 22:56:13.618450 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:13.618453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:13.618463 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:56:13.618550 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:13.618551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:13.618561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:13.618612 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:13.618786 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.619630 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.621057 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.621748 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.621822 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.621831 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:13.622008 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 22:56:13.622047 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 22:56:13.622164 28984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.618348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37617 (local address 127.14.14.130:55913)
0928 22:56:13.618401 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:13.618416 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:13.618466 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:13.618642 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:13.618642 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:13.618645 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:13.618718 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.618719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.619717 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.619717 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.620954 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.620957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.621835 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.621836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.621917 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.621925 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 22:56:13.621985 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:56:13.622037 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":17}
I20260928 22:56:13.622398 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.618427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:39707 (local address 127.14.14.130:51221)
0928 22:56:13.618594 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:13.618604 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:13.618653 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:13.618852 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:13.618853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:13.618854 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:13.618907 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.618908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.619782 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.619783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.620966 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:13.620968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.621836 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.621836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:13.621917 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.621925 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 22:56:13.621985 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:56:13.622037 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":44,"thread_start_us":120,"threads_started":1}
I20260928 22:56:13.622601 29068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29534103ecf5486d9ff523c1855ae8af" candidate_uuid: "c3689bf232744d9cbec0b595e533deef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" is_pre_election: true
I20260928 22:56:13.622815 29068 raft_consensus.cc:2454] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3689bf232744d9cbec0b595e533deef in term 0.
I20260928 22:56:13.622838 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29534103ecf5486d9ff523c1855ae8af" candidate_uuid: "c3689bf232744d9cbec0b595e533deef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9ba290a5c3841cbb20f14d27b800057" is_pre_election: true
I20260928 22:56:13.623003 28792 raft_consensus.cc:2454] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3689bf232744d9cbec0b595e533deef in term 0.
I20260928 22:56:13.623088 28871 leader_election.cc:302] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 072909ddd02b4be4b0fb286ddd99eeaf, c3689bf232744d9cbec0b595e533deef; no voters:
I20260928 22:56:13.623212 29128 raft_consensus.cc:2781] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:56:13.623267 29128 raft_consensus.cc:482] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:56:13.623311 29128 raft_consensus.cc:3037] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:13.624017 29128 raft_consensus.cc:504] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:13.624163 29128 leader_election.cc:288] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [CANDIDATE]: Term 1 election: Requested vote from peers 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617), b9ba290a5c3841cbb20f14d27b800057 (127.14.14.129:39707)
I20260928 22:56:13.624285 29068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29534103ecf5486d9ff523c1855ae8af" candidate_uuid: "c3689bf232744d9cbec0b595e533deef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "072909ddd02b4be4b0fb286ddd99eeaf"
I20260928 22:56:13.624377 29068 raft_consensus.cc:3037] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:13.624408 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29534103ecf5486d9ff523c1855ae8af" candidate_uuid: "c3689bf232744d9cbec0b595e533deef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9ba290a5c3841cbb20f14d27b800057"
I20260928 22:56:13.624526 28792 raft_consensus.cc:3037] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:13.625247 29068 raft_consensus.cc:2454] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3689bf232744d9cbec0b595e533deef in term 1.
I20260928 22:56:13.625381 28871 leader_election.cc:302] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 072909ddd02b4be4b0fb286ddd99eeaf, c3689bf232744d9cbec0b595e533deef; no voters:
I20260928 22:56:13.625484 29128 raft_consensus.cc:2781] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:56:13.625545 28792 raft_consensus.cc:2454] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3689bf232744d9cbec0b595e533deef in term 1.
I20260928 22:56:13.625670 29128 raft_consensus.cc:686] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [term 1 LEADER]: Becoming Leader. State: Replica: c3689bf232744d9cbec0b595e533deef, State: Running, Role: LEADER
I20260928 22:56:13.625778 29128 consensus_queue.cc:237] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:13.626621 28656 catalog_manager.cc:5495] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef reported cstate change: term changed from 0 to 1, leader changed from <none> to c3689bf232744d9cbec0b595e533deef (127.14.14.130). New cstate: current_term: 1 leader_uuid: "c3689bf232744d9cbec0b595e533deef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } health_report { overall_health: UNKNOWN } } }
I20260928 22:56:13.649369 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.646586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.14.14.130:43083)
0928 22:56:13.646648 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:13.646651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:13.646661 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:56:13.646705 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:13.646706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:13.646727 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:13.646819 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:56:13.646951 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.647496 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.648408 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:13.648676 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:13.648761 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:13.648949 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:56:13.649096 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:56:13.649193 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:13.649233 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 22:56:13.649254 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:56:13.651502 29068 raft_consensus.cc:1261] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf [term 1 FOLLOWER]: Refusing update from remote peer c3689bf232744d9cbec0b595e533deef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:56:13.651505 28792 raft_consensus.cc:1261] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Refusing update from remote peer c3689bf232744d9cbec0b595e533deef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:56:13.651819 29128 consensus_queue.cc:1035] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:56:13.651975 29131 consensus_queue.cc:1035] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [LEADER]: Connected to new peer: Peer: permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:56:13.655426 29136 mvcc.cc:205] Tried to move back new op lower bound from 7334445767273537536 to 7334445767171874816. Current Snapshot: MvccSnapshot[applied={T|T < 7334445767273537536 or (T in {7334445767273537536})}]
I20260928 22:56:13.656796 28655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:56:13.656890 28655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:56:13.657774 28655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: Sending DeleteTablet for 3 replicas of tablet 29534103ecf5486d9ff523c1855ae8af
I20260928 22:56:13.658107 28916 tablet_service.cc:1514] Processing DeleteTablet for tablet 29534103ecf5486d9ff523c1855ae8af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:13 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:39972
I20260928 22:56:13.658161 29048 tablet_service.cc:1514] Processing DeleteTablet for tablet 29534103ecf5486d9ff523c1855ae8af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:13 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:45826
I20260928 22:56:13.658187 28778 tablet_service.cc:1514] Processing DeleteTablet for tablet 29534103ecf5486d9ff523c1855ae8af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:13 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:48586
I20260928 22:56:13.658277 28853 tablet_replica.cc:330] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057: stopping tablet replica
I20260928 22:56:13.658324 28853 raft_consensus.cc:2229] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:56:13.658384 28853 raft_consensus.cc:2258] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:56:13.658409 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 28694
I20260928 22:56:13.658593 29144 tablet_replica.cc:330] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef: stopping tablet replica
I20260928 22:56:13.658711 29144 raft_consensus.cc:2229] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:56:13.658844 29144 raft_consensus.cc:2258] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:56:13.659258 29144 ts_tablet_manager.cc:1905] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:56:13.660981 29144 ts_tablet_manager.cc:1918] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:56:13.661077 29144 log.cc:1196] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/29534103ecf5486d9ff523c1855ae8af
I20260928 22:56:13.661389 29144 ts_tablet_manager.cc:1939] T 29534103ecf5486d9ff523c1855ae8af P c3689bf232744d9cbec0b595e533deef: Deleting consensus metadata
I20260928 22:56:13.661856 28643 catalog_manager.cc:4841] TS c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083): tablet 29534103ecf5486d9ff523c1855ae8af (table test-table [id=3f9be8d67ead4cdcb11d72c1e16c51e4]) successfully deleted
I20260928 22:56:13.665781 29145 tablet_replica.cc:330] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf: stopping tablet replica
W20260928 22:56:13.665951 28641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:39707 (error 108)
I20260928 22:56:13.666004 29145 raft_consensus.cc:2229] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:56:13.666404 29145 raft_consensus.cc:2258] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:56:13.666534 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:13.666409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:39707 (local address 127.0.0.1:48590)
0928 22:56:13.666465 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:13.666501 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:39707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 22:56:13.666627 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 28849
W20260928 22:56:13.666648 28641 catalog_manager.cc:4584] TS b9ba290a5c3841cbb20f14d27b800057 (127.14.14.129:39707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29534103ecf5486d9ff523c1855ae8af: Network error: Client connection negotiation failed: client connection to 127.14.14.129:39707: connect: Connection refused (error 111)
I20260928 22:56:13.668354 29145 ts_tablet_manager.cc:1905] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:56:13.670101 29145 ts_tablet_manager.cc:1918] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:56:13.670202 29145 log.cc:1196] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/29534103ecf5486d9ff523c1855ae8af
I20260928 22:56:13.670459 29145 ts_tablet_manager.cc:1939] T 29534103ecf5486d9ff523c1855ae8af P 072909ddd02b4be4b0fb286ddd99eeaf: Deleting consensus metadata
I20260928 22:56:13.670902 28642 catalog_manager.cc:4841] TS 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617): tablet 29534103ecf5486d9ff523c1855ae8af (table test-table [id=3f9be8d67ead4cdcb11d72c1e16c51e4]) successfully deleted
I20260928 22:56:13.672108 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 28985
I20260928 22:56:13.676783 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 28623
I20260928 22:56:13.681890 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44663
--webserver_interface=127.14.14.190
--webserver_port=44509
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:44663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:13.764087 29147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:13.764304 29147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:13.764328 29147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:13.765446 29147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:56:13.765482 29147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:13.765497 29147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:13.765509 29147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:56:13.765522 29147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:56:13.767043 29147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:44663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=44509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:13.767241 29147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:13.767411 29147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:13.772464 29153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:13.772469 29152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:13.772677 29147 server_base.cc:1034] running on GCE node
W20260928 22:56:13.772469 29155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:13.772986 29147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:13.773289 29147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:13.774434 29147 hybrid_clock.cc:648] HybridClock initialized: now 1790636173774408 us; error 54 us; skew 500 ppm
Sep 28 22:56:13 dist-test-slave-kzfr krb5kdc[28074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:13.776997 29147 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:56:13.777477 29147 webserver.cc:460] Webserver started at http://127.14.14.190:44509/ using document root <none> and password file <none>
I20260928 22:56:13.777766 29147 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:13.777840 29147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:13.779191 29147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:13.780035 29162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:13.780294 29147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:13.780411 29147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "d89d2420acf6402cbba1f9fa04f70fa0"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "a1191e486fd9c8c723cc139fb1d9090e"
server_key_iv: "299d5b1ef61c70873ba3e2b90ebfaec2"
server_key_version: "encryptionkey@0"
I20260928 22:56:13.780771 29147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:13.794391 29147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:13.796504 29147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:13.796669 29147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:13.800890 29147 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:44663
I20260928 22:56:13.800925 29214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:44663 every 8 connection(s)
I20260928 22:56:13.801353 29147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:56:13.803779 29215 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:56:13.804795 29215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: Bootstrap starting.
I20260928 22:56:13.806840 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 29147
I20260928 22:56:13.807133 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:39707
--local_ip_for_outbound_sockets=127.14.14.129
--tserver_master_addrs=127.14.14.190:44663
--webserver_port=34333
--webserver_interface=127.14.14.129
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:56:13.815294 29215 log.cc:824] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:13.817797 29215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:56:13.818125 29215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0: Bootstrap complete.
I20260928 22:56:13.820415 29215 raft_consensus.cc:348] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } }
I20260928 22:56:13.820637 29215 raft_consensus.cc:729] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d89d2420acf6402cbba1f9fa04f70fa0, State: Initialized, Role: FOLLOWER
I20260928 22:56:13.820816 29215 consensus_queue.cc:260] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } }
I20260928 22:56:13.820889 29215 raft_consensus.cc:388] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:56:13.820959 29215 raft_consensus.cc:482] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:56:13.821007 29215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 2 FOLLOWER]: Advancing to term 3
I20260928 22:56:13.822184 29215 raft_consensus.cc:504] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } }
I20260928 22:56:13.822316 29215 leader_election.cc:302] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d89d2420acf6402cbba1f9fa04f70fa0; no voters:
I20260928 22:56:13.822536 29215 leader_election.cc:288] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 22:56:13.822572 29219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 22:56:13.822681 29219 raft_consensus.cc:686] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [term 3 LEADER]: Becoming Leader. State: Replica: d89d2420acf6402cbba1f9fa04f70fa0, State: Running, Role: LEADER
I20260928 22:56:13.822786 29219 consensus_queue.cc:237] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } }
I20260928 22:56:13.822824 29215 sys_catalog.cc:564] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:56:13.823067 29221 sys_catalog.cc:455] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } } }
I20260928 22:56:13.823326 29221 sys_catalog.cc:458] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:13.823261 29220 sys_catalog.cc:455] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d89d2420acf6402cbba1f9fa04f70fa0. Latest consensus state: current_term: 3 leader_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89d2420acf6402cbba1f9fa04f70fa0" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44663 } } }
I20260928 22:56:13.823612 29220 sys_catalog.cc:458] T 00000000000000000000000000000000 P d89d2420acf6402cbba1f9fa04f70fa0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:13.823688 29228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:56:13.824604 29228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:56:13.824725 29228 catalog_manager.cc:1259] Loaded cluster ID: 84e1c2f10825462297fa671b65491815
I20260928 22:56:13.824939 29228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 22:56:13.825161 29228 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff7938369ad google::LogMessage::Fail() at ??:0
@ 0x7ff79383988a google::LogMessage::SendToLog() at ??:0
@ 0x7ff7938364de google::LogMessage::Flush() at ??:0
@ 0x7ff793837a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff79b7570f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff79b7608b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff7957040d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff7956f9954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff79655d6db start_thread at ??:0
@ 0x7ff79233461f clone at ??:0
W20260928 22:56:13.907439 29217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:13.907657 29217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:13.907685 29217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:13.908864 29217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:13.908906 29217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:13.908933 29217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:56:13.910545 29217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:39707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=34333
--tserver_master_addrs=127.14.14.190:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:13.910796 29217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:13.911379 29217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:13.917482 29252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:13.917478 29251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:13.917478 29254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:13.918149 29217 server_base.cc:1034] running on GCE node
I20260928 22:56:13.918427 29217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:13.918725 29217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:13.919917 29217 hybrid_clock.cc:648] HybridClock initialized: now 1790636173919899 us; error 35 us; skew 500 ppm
Sep 28 22:56:13 dist-test-slave-kzfr krb5kdc[28074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:13.922586 29217 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:56:13.923054 29217 webserver.cc:460] Webserver started at http://127.14.14.129:34333/ using document root <none> and password file <none>
I20260928 22:56:13.923300 29217 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:13.923382 29217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:13.924791 29217 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:13.925835 29261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:13.926106 29217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:13.926218 29217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "b9ba290a5c3841cbb20f14d27b800057"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "77181d70b3de5e05a975c035af9a3ecd"
server_key_iv: "4938a71f89fba0e3f01c2778ffbad8a7"
server_key_version: "encryptionkey@0"
I20260928 22:56:13.926620 29217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:13.938328 29217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:13.941066 29217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:13.941320 29217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:13.941663 29217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:13.942456 29268 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:56:13.944716 29217 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:56:13.944824 29217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260928 22:56:13.944875 29217 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:56:13.945549 29217 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:56:13.945633 29217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260928 22:56:13.945712 29268 tablet_bootstrap.cc:492] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057: Bootstrap starting.
I20260928 22:56:13.953240 29217 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:39707
I20260928 22:56:13.953305 29375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:39707 every 8 connection(s)
I20260928 22:56:13.953727 29217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:56:13.954273 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 29217
I20260928 22:56:13.954562 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:43083
--local_ip_for_outbound_sockets=127.14.14.130
--tserver_master_addrs=127.14.14.190:44663
--webserver_port=44085
--webserver_interface=127.14.14.130
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:56:13.957926 29268 log.cc:824] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:13.960939 29268 tablet_bootstrap.cc:492] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:56:13.961279 29268 tablet_bootstrap.cc:492] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057: Bootstrap complete.
I20260928 22:56:13.962502 29268 ts_tablet_manager.cc:1397] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.011s
I20260928 22:56:13.964093 29268 raft_consensus.cc:348] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:13.964493 29268 raft_consensus.cc:729] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ba290a5c3841cbb20f14d27b800057, State: Initialized, Role: FOLLOWER
I20260928 22:56:13.964656 29268 consensus_queue.cc:260] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:13.964962 29268 ts_tablet_manager.cc:1428] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260928 22:56:14.036502 29379 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:56:13.956540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:58999)
0928 22:56:13.956949 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:13.956995 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:13.957804 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:14.036260 (+ 78456us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":92,"threads_started":1}
W20260928 22:56:14.036779 29376 heartbeater.cc:644] Failed to heartbeat to 127.14.14.190:44663 (0 consecutive failures): Network error: Failed to ping master at 127.14.14.190:44663: Client connection negotiation failed: client connection to 127.14.14.190:44663: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 22:56:14.037196 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:14.037068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:58231)
0928 22:56:14.037129 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:14.037174 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 22:56:14.037657 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:14.037526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:49419)
0928 22:56:14.037574 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:14.037635 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 22:56:14.037796 29376 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:56:14.041813 29378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:14.042030 29378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:14.042081 29378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:14.043349 29378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:14.043416 29378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:14.043467 29378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:56:14.045234 29378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:43083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=44085
--tserver_master_addrs=127.14.14.190:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:14.045511 29378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:14.045778 29378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:14.051182 29386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:14.051174 29387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:14.051174 29389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:14.051512 29378 server_base.cc:1034] running on GCE node
I20260928 22:56:14.051818 29378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:14.052117 29378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:14.053314 29378 hybrid_clock.cc:648] HybridClock initialized: now 1790636174053290 us; error 37 us; skew 500 ppm
Sep 28 22:56:14 dist-test-slave-kzfr krb5kdc[28074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636174, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:14.056262 29378 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:56:14.056777 29378 webserver.cc:460] Webserver started at http://127.14.14.130:44085/ using document root <none> and password file <none>
I20260928 22:56:14.056989 29378 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:14.057032 29378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:14.058554 29378 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:56:14.059172 29396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:14.059412 29378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:56:14.059499 29378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "c3689bf232744d9cbec0b595e533deef"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "d4b22cbd1e5a709b52335794a0189d6b"
server_key_iv: "0f614319f386051befa22c43ec2991ac"
server_key_version: "encryptionkey@0"
I20260928 22:56:14.059829 29378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:14.086272 29378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:14.088640 29378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:14.088773 29378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:14.088977 29378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:14.089340 29378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:14.089375 29378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:14.089435 29378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:14.089468 29378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:14.095543 29378 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:43083
I20260928 22:56:14.095587 29509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:43083 every 8 connection(s)
I20260928 22:56:14.096021 29378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:56:14.097103 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:14.096698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:52619)
0928 22:56:14.096875 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:14.096954 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
W20260928 22:56:14.097561 29510 heartbeater.cc:644] Failed to heartbeat to 127.14.14.190:44663 (0 consecutive failures): Network error: Failed to ping master at 127.14.14.190:44663: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
I20260928 22:56:14.097903 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:14.097799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:51339)
0928 22:56:14.097852 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:14.097879 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:56:14.098145 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:14.098094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:59669)
0928 22:56:14.098114 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:14.098132 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 22:56:14.098304 29510 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:56:14.100603 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 29378
I20260928 22:56:14.100869 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:37617
--local_ip_for_outbound_sockets=127.14.14.131
--tserver_master_addrs=127.14.14.190:44663
--webserver_port=46117
--webserver_interface=127.14.14.131
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:14.182710 29513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:14.182914 29513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:14.182937 29513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:14.184087 29513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:14.184130 29513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:14.184155 29513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:56:14.185613 29513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:37617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=46117
--tserver_master_addrs=127.14.14.190:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:14.185913 29513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:14.186131 29513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:14.191336 29519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:14.191396 29521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:14.191429 29518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:14.191610 29513 server_base.cc:1034] running on GCE node
I20260928 22:56:14.191797 29513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:14.191995 29513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:14.193135 29513 hybrid_clock.cc:648] HybridClock initialized: now 1790636174193074 us; error 75 us; skew 500 ppm
Sep 28 22:56:14 dist-test-slave-kzfr krb5kdc[28074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636174, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:14.195753 29513 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:56:14.196249 29513 webserver.cc:460] Webserver started at http://127.14.14.131:46117/ using document root <none> and password file <none>
I20260928 22:56:14.196489 29513 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:14.196563 29513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:14.197984 29513 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:56:14.198594 29528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:14.198772 29513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:56:14.198874 29513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "072909ddd02b4be4b0fb286ddd99eeaf"
format_stamp: "Formatted at 2026-09-28 22:56:12 on dist-test-slave-kzfr"
server_key: "08744b3a71e7a5f155b4864b6fdae481"
server_key_iv: "835821033214daff91467e2f4adaca58"
server_key_version: "encryptionkey@0"
I20260928 22:56:14.199226 29513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:14.209369 29513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:14.211831 29513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:14.212002 29513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:14.212239 29513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:14.212559 29513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:14.212616 29513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:14.212658 29513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:14.212687 29513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:14.219233 29513 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:37617
I20260928 22:56:14.219290 29641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:37617 every 8 connection(s)
I20260928 22:56:14.219666 29513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:56:14.220734 29644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:14.220351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:41465)
0928 22:56:14.220517 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:14.220595 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
W20260928 22:56:14.221174 29642 heartbeater.cc:644] Failed to heartbeat to 127.14.14.190:44663 (0 consecutive failures): Network error: Failed to ping master at 127.14.14.190:44663: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
I20260928 22:56:14.221457 29644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:14.221368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:39589)
0928 22:56:14.221413 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:14.221439 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 22:56:14.221776 29644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:14.221701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:37953)
0928 22:56:14.221741 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:14.221763 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 22:56:14.221899 29642 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:56:14.225785 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 29513
I20260928 22:56:15.038465 29646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:15.038170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:45967)
0928 22:56:15.038363 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:15.038400 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":127,"threads_started":1}
I20260928 22:56:15.098816 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:15.098612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:50777)
0928 22:56:15.098722 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:15.098756 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":51,"threads_started":1}
I20260928 22:56:15.222474 29648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:15.222198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:43175)
0928 22:56:15.222393 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:15.222423 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":124,"threads_started":1}
I20260928 22:56:15.773732 29649 raft_consensus.cc:482] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:56:15.773854 29649 raft_consensus.cc:504] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:15.774241 29649 leader_election.cc:288] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617), c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083)
Sep 28 22:56:15 dist-test-slave-kzfr krb5kdc[28074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.131@KRBTEST.COM
Sep 28 22:56:15 dist-test-slave-kzfr krb5kdc[28074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.130@KRBTEST.COM
I20260928 22:56:15.788039 29653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:15.774625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:58771 (local address 127.14.14.130:43083)
0928 22:56:15.774818 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:15.774826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:15.774845 (+ 19us) server_negotiation.cc:407] Connection header received
0928 22:56:15.775386 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:15.775389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:15.775451 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:15.775604 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:15.776809 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:15.778260 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:15.779936 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:15.780575 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:15.780708 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:15.784119 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:15.784123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:15.784123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:15.784132 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:15.786034 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:15.786589 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:15.786592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:15.786593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:15.786694 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:15.787086 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:15.787088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:15.787088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:15.787416 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:15.787511 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:15.787783 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 22:56:15.787920 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260928 22:56:15.788029 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:15.774486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37617 (local address 127.14.14.129:57485)
0928 22:56:15.774625 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:15.774637 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:15.774698 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:15.775718 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:15.775721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:15.776346 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:15.776653 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:15.776662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:15.778436 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:15.778437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:15.779812 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:15.779814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:15.780716 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:15.780717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:15.780856 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:15.781701 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:15.781712 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:15.783607 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:15.786017 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:15.786023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:15.786026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:15.786491 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:15.786935 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:15.786936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:15.786936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:15.787018 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:15.787519 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:15.787520 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:15.787617 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 22:56:15.787731 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 22:56:15.787932 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":101,"thread_start_us":52,"threads_started":1}
I20260928 22:56:15.788057 29652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:15.774588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:57485 (local address 127.14.14.131:37617)
0928 22:56:15.774763 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:15.774778 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:15.774802 (+ 24us) server_negotiation.cc:407] Connection header received
0928 22:56:15.775477 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:15.775480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:15.775524 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:15.775637 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:15.776809 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:15.778341 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:15.779936 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:15.780620 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:15.780712 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:15.783722 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:15.783726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:15.783726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:15.783736 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:15.785932 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:15.786625 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:15.786628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:15.786628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:15.786846 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:15.787113 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:15.787118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:15.787119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:15.787423 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:15.787520 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:15.787786 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 22:56:15.787920 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":68,"threads_started":1}
I20260928 22:56:15.788553 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:15.774486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:43083 (local address 127.14.14.129:58771)
0928 22:56:15.774600 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:15.774613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:15.774694 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:15.775718 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:15.775721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:15.776346 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:15.776653 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:15.776662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:15.778420 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:15.778421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:15.779812 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:15.779814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:15.780731 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:15.780732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:15.780856 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:15.781740 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:15.781744 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:15.783999 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:15.786158 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:15.786159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:15.786159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:15.786485 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:15.786884 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:15.786885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:15.786886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:15.787015 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:15.787491 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:15.787497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:15.787622 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 22:56:15.787732 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 22:56:15.788015 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":224,"thread_start_us":70,"threads_started":1}
W20260928 22:56:15.788965 29264 leader_election.cc:341] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617): Not found: Tablet not found: 29534103ecf5486d9ff523c1855ae8af
W20260928 22:56:15.789237 29265 leader_election.cc:341] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083): Not found: Tablet not found: 29534103ecf5486d9ff523c1855ae8af
I20260928 22:56:15.789278 29265 leader_election.cc:302] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9ba290a5c3841cbb20f14d27b800057; no voters: 072909ddd02b4be4b0fb286ddd99eeaf, c3689bf232744d9cbec0b595e533deef
I20260928 22:56:15.789407 29649 raft_consensus.cc:2726] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:56:16.039275 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:16.039096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:35463)
0928 22:56:16.039182 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:16.039226 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 22:56:16.099529 29654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:16.099279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:37913)
0928 22:56:16.099440 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:16.099484 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260928 22:56:16.223345 29655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:16.223082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:40015)
0928 22:56:16.223241 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:16.223297 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260928 22:56:17.039985 29656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:17.039768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:50465)
0928 22:56:17.039902 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:17.039939 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260928 22:56:17.100293 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:17.100021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:57531)
0928 22:56:17.100195 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:17.100247 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260928 22:56:17.224215 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:17.223948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:39949)
0928 22:56:17.224109 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:17.224156 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
I20260928 22:56:17.523278 29659 raft_consensus.cc:482] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:56:17.523413 29659 raft_consensus.cc:504] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:17.523638 29659 leader_election.cc:288] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617), c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083)
W20260928 22:56:17.524125 29265 leader_election.cc:341] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083): Not found: Tablet not found: 29534103ecf5486d9ff523c1855ae8af
W20260928 22:56:17.524199 29264 leader_election.cc:341] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617): Not found: Tablet not found: 29534103ecf5486d9ff523c1855ae8af
I20260928 22:56:17.524238 29264 leader_election.cc:302] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9ba290a5c3841cbb20f14d27b800057; no voters: 072909ddd02b4be4b0fb286ddd99eeaf, c3689bf232744d9cbec0b595e533deef
I20260928 22:56:17.524313 29659 raft_consensus.cc:2726] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:56:18.040874 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:18.040583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:45041)
0928 22:56:18.040766 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:18.040825 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":64,"threads_started":1}
I20260928 22:56:18.100963 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:18.100751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:46631)
0928 22:56:18.100885 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:18.100923 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260928 22:56:18.224979 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:18.224750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:52383)
0928 22:56:18.224894 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:18.224934 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
I20260928 22:56:19.041919 29663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:19.041571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:48165)
0928 22:56:19.041823 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:19.041872 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
I20260928 22:56:19.101773 29664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:19.101549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:45383)
0928 22:56:19.101684 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:19.101727 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260928 22:56:19.225716 29665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:19.225421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:43375)
0928 22:56:19.225565 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:19.225661 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":58,"threads_started":1}
I20260928 22:56:20.042737 29666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:20.042521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:43159)
0928 22:56:20.042655 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:20.042690 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260928 22:56:20.059405 29667 raft_consensus.cc:482] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:56:20.059504 29667 raft_consensus.cc:504] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:20.059763 29667 leader_election.cc:288] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617), c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083)
W20260928 22:56:20.060176 29264 leader_election.cc:341] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617): Not found: Tablet not found: 29534103ecf5486d9ff523c1855ae8af
W20260928 22:56:20.060328 29265 leader_election.cc:341] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083): Not found: Tablet not found: 29534103ecf5486d9ff523c1855ae8af
I20260928 22:56:20.060403 29265 leader_election.cc:302] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9ba290a5c3841cbb20f14d27b800057; no voters: 072909ddd02b4be4b0fb286ddd99eeaf, c3689bf232744d9cbec0b595e533deef
I20260928 22:56:20.060483 29667 raft_consensus.cc:2726] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:56:20.102488 29668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:20.102239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:60577)
0928 22:56:20.102398 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:20.102448 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":61,"threads_started":1}
I20260928 22:56:20.226436 29669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:20.226218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:54965)
0928 22:56:20.226355 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:20.226393 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260928 22:56:21.043586 29670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:21.043348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:32847)
0928 22:56:21.043501 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:21.043538 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260928 22:56:21.103201 29671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:21.102984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:37311)
0928 22:56:21.103119 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:21.103160 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
I20260928 22:56:21.227169 29672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:21.226942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:35339)
0928 22:56:21.227088 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:21.227126 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260928 22:56:22.044431 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:22.044142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:51485)
0928 22:56:22.044347 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:22.044385 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":65,"threads_started":1}
I20260928 22:56:22.103976 29674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:22.103744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:45813)
0928 22:56:22.103892 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:22.103931 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260928 22:56:22.227918 29675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:22.227700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:39063)
0928 22:56:22.227833 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:22.227875 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260928 22:56:23.045476 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:23.045217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:53259)
0928 22:56:23.045397 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:23.045429 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":95,"threads_started":1}
I20260928 22:56:23.104710 29677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:23.104501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:42857)
0928 22:56:23.104624 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:23.104665 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260928 22:56:23.228823 29678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:23.228578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:60637)
0928 22:56:23.228719 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:23.228761 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":58,"threads_started":1}
I20260928 22:56:24.046219 29679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:24.045976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:46965)
0928 22:56:24.046131 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:24.046169 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20260928 22:56:24.105388 29680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:24.105157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:51507)
0928 22:56:24.105306 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:24.105345 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260928 22:56:24.229562 29681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:24.229334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:51325)
0928 22:56:24.229475 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:24.229515 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260928 22:56:24.565116 29682 raft_consensus.cc:482] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:56:24.565232 29682 raft_consensus.cc:504] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:24.565486 29682 leader_election.cc:288] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617), c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083)
W20260928 22:56:24.565948 29264 leader_election.cc:341] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617): Not found: Tablet not found: 29534103ecf5486d9ff523c1855ae8af
W20260928 22:56:24.566038 29265 leader_election.cc:341] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083): Not found: Tablet not found: 29534103ecf5486d9ff523c1855ae8af
I20260928 22:56:24.566115 29265 leader_election.cc:302] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9ba290a5c3841cbb20f14d27b800057; no voters: 072909ddd02b4be4b0fb286ddd99eeaf, c3689bf232744d9cbec0b595e533deef
I20260928 22:56:24.566213 29682 raft_consensus.cc:2726] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:56:25.046974 29683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:25.046729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:54077)
0928 22:56:25.046889 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:25.046926 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":66,"threads_started":1}
I20260928 22:56:25.106097 29684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:25.105894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:53549)
0928 22:56:25.106015 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:25.106053 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260928 22:56:25.230403 29685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:25.230169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:59107)
0928 22:56:25.230313 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:25.230353 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260928 22:56:26.047776 29686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:26.047559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:36571)
0928 22:56:26.047692 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:26.047729 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260928 22:56:26.106875 29687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:26.106648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:33271)
0928 22:56:26.106786 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:26.106830 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260928 22:56:26.231120 29688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:26.230905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:42473)
0928 22:56:26.231037 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:26.231075 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260928 22:56:26.285852 29689 raft_consensus.cc:482] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:56:26.285988 29689 raft_consensus.cc:504] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072909ddd02b4be4b0fb286ddd99eeaf" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37617 } } peers { permanent_uuid: "c3689bf232744d9cbec0b595e533deef" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43083 } } peers { permanent_uuid: "b9ba290a5c3841cbb20f14d27b800057" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 39707 } }
I20260928 22:56:26.286211 29689 leader_election.cc:288] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617), c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083)
W20260928 22:56:26.286643 29264 leader_election.cc:341] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 072909ddd02b4be4b0fb286ddd99eeaf (127.14.14.131:37617): Not found: Tablet not found: 29534103ecf5486d9ff523c1855ae8af
W20260928 22:56:26.286728 29265 leader_election.cc:341] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3689bf232744d9cbec0b595e533deef (127.14.14.130:43083): Not found: Tablet not found: 29534103ecf5486d9ff523c1855ae8af
I20260928 22:56:26.286803 29265 leader_election.cc:302] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9ba290a5c3841cbb20f14d27b800057; no voters: 072909ddd02b4be4b0fb286ddd99eeaf, c3689bf232744d9cbec0b595e533deef
I20260928 22:56:26.286873 29689 raft_consensus.cc:2726] T 29534103ecf5486d9ff523c1855ae8af P b9ba290a5c3841cbb20f14d27b800057 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:56:27.048462 29690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:27.048232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:54067)
0928 22:56:27.048378 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:27.048415 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
I20260928 22:56:27.107632 29691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:27.107401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:43301)
0928 22:56:27.107553 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:27.107591 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":62,"threads_started":1}
I20260928 22:56:27.231812 29692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:27.231583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:38175)
0928 22:56:27.231721 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:27.231761 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":59,"threads_started":1}
I20260928 22:56:28.049335 29693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:28.049010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:33237)
0928 22:56:28.049182 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:28.049216 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260928 22:56:28.108438 29694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:28.108179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:56521)
0928 22:56:28.108325 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:28.108393 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260928 22:56:28.232508 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:28.232275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:50057)
0928 22:56:28.232423 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:28.232463 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260928 22:56:29.050261 29696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:29.050039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.129:38613)
0928 22:56:29.050171 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:29.050210 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260928 22:56:29.109258 29697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:29.109042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.130:36581)
0928 22:56:29.109172 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:29.109211 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260928 22:56:29.232066 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 29217
I20260928 22:56:29.233836 29698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:29.233463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44663 (local address 127.14.14.131:55759)
0928 22:56:29.233749 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:29.233795 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":53,"threads_started":1}
I20260928 22:56:29.237044 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 29378
I20260928 22:56:29.241384 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 29513
2026-09-28T22:56:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18518 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:56:29 dist-test-slave-kzfr krb5kdc[29701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:56:29 dist-test-slave-kzfr krb5kdc[29701](info): set up 2 sockets
Sep 28 22:56:29 dist-test-slave-kzfr krb5kdc[29701](info): commencing operation
krb5kdc: starting...
W20260928 22:56:30.286641 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:56:30 dist-test-slave-kzfr krb5kdc[29701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:56:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:56:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:56:30.413546 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:33333
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:33333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:30.497501 29714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:30.497852 29714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:30.497896 29714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:30.499029 29714 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 22:56:30.499064 29714 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 22:56:30.499078 29714 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 22:56:30.499089 29714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:56:30.499102 29714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:30.499115 29714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:30.499128 29714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:56:30.499140 29714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:56:30.500535 29714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:33333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:33333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:30.500725 29714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:30.500902 29714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:30.505756 29719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:30.505797 29720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:30.505826 29722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:30.505972 29714 server_base.cc:1034] running on GCE node
I20260928 22:56:30.506229 29714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:30.506536 29714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:30.507702 29714 hybrid_clock.cc:648] HybridClock initialized: now 1790636190507685 us; error 50 us; skew 500 ppm
Sep 28 22:56:30 dist-test-slave-kzfr krb5kdc[29701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636190, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:30.510350 29714 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:56:30.510811 29714 webserver.cc:460] Webserver started at http://127.14.14.190:44091/ using document root <none> and password file <none>
I20260928 22:56:30.511037 29714 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:30.511106 29714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:30.511262 29714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:56:30.512301 29714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "24fd1c1861fc4832a7c8cb28644b36b4"
format_stamp: "Formatted at 2026-09-28 22:56:30 on dist-test-slave-kzfr"
server_key: "75004178183ce921b2cdf59a201d5599"
server_key_iv: "b50f83b0745618f2974c78bcf8e2f21c"
server_key_version: "encryptionkey@0"
I20260928 22:56:30.512681 29714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "24fd1c1861fc4832a7c8cb28644b36b4"
format_stamp: "Formatted at 2026-09-28 22:56:30 on dist-test-slave-kzfr"
server_key: "75004178183ce921b2cdf59a201d5599"
server_key_iv: "b50f83b0745618f2974c78bcf8e2f21c"
server_key_version: "encryptionkey@0"
I20260928 22:56:30.514066 29714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:56:30.514791 29729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:30.514953 29714 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:56:30.515056 29714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "24fd1c1861fc4832a7c8cb28644b36b4"
format_stamp: "Formatted at 2026-09-28 22:56:30 on dist-test-slave-kzfr"
server_key: "75004178183ce921b2cdf59a201d5599"
server_key_iv: "b50f83b0745618f2974c78bcf8e2f21c"
server_key_version: "encryptionkey@0"
I20260928 22:56:30.515129 29714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:30.546110 29714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:30.548122 29714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:30.548292 29714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:30.552498 29714 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:33333
I20260928 22:56:30.552537 29781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:33333 every 8 connection(s)
I20260928 22:56:30.552917 29714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:56:30.553992 29782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:30.556345 29782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Bootstrap starting.
I20260928 22:56:30.556926 29782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:30.557221 29782 log.cc:824] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:30.558012 29782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: No bootstrap required, opened a new log
I20260928 22:56:30.558743 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 29714
I20260928 22:56:30.558830 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:56:30.559036 14394 external_mini_cluster.cc:1428] Setting key 5f2a6b523216c30b98e7dfb00a377fb3
I20260928 22:56:30.560226 29782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } }
I20260928 22:56:30.560408 29782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:30.560447 29782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd1c1861fc4832a7c8cb28644b36b4, State: Initialized, Role: FOLLOWER
I20260928 22:56:30.560599 29782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } }
I20260928 22:56:30.560663 29782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:56:30.560683 29782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:56:30.560743 29782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:30.561759 29782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } }
I20260928 22:56:30.561868 29782 leader_election.cc:302] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24fd1c1861fc4832a7c8cb28644b36b4; no voters:
I20260928 22:56:30.562016 29782 leader_election.cc:288] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:56:30.562083 29787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:56:30.562250 29782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:56:30.562273 29787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 1 LEADER]: Becoming Leader. State: Replica: 24fd1c1861fc4832a7c8cb28644b36b4, State: Running, Role: LEADER
I20260928 22:56:30.562517 29787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } }
I20260928 22:56:30.562932 29788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } } }
I20260928 22:56:30.563007 29788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:30.563244 29789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24fd1c1861fc4832a7c8cb28644b36b4. Latest consensus state: current_term: 1 leader_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } } }
I20260928 22:56:30.563386 29796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:56:30.564067 29796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:56:30.564605 29789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:30.565752 29796 catalog_manager.cc:1347] Generated new cluster ID: 77119e044c8448a6a17453431774715c
I20260928 22:56:30.565817 29796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:56:30 dist-test-slave-kzfr krb5kdc[29701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:30.570444 29786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:30.559592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57780 (local address 127.14.14.190:33333)
0928 22:56:30.559814 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:30.559824 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:30.559859 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:56:30.560177 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:30.560185 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:30.560246 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:30.560395 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:30.561183 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:30.562734 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:30.563874 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:30.564460 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:30.564568 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:30.567183 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:30.567187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:30.567188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:30.567199 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:30.568874 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:30.569255 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:30.569258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:30.569259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:30.569366 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:30.569577 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:30.569599 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:30.569599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:30.569874 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:30.570026 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:30.570139 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 22:56:30.570280 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":134,"threads_started":1}
I20260928 22:56:30.601302 29796 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:56:30.602063 29796 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:56:30.605994 29796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Generated new TSK 0
I20260928 22:56:30.606187 29796 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:56:30.640345 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:33333
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:30.734751 29808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:30.734965 29808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:30.734989 29808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:30.736123 29808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:30.736163 29808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:30.736189 29808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:56:30.737758 29808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:33333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:30.738026 29808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:30.738242 29808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:30.743199 29813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:30.743211 29814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:30.743315 29808 server_base.cc:1034] running on GCE node
W20260928 22:56:30.743211 29816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:30.743639 29808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:30.743891 29808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:30.745049 29808 hybrid_clock.cc:648] HybridClock initialized: now 1790636190745045 us; error 35 us; skew 500 ppm
Sep 28 22:56:30 dist-test-slave-kzfr krb5kdc[29701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636190, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:30.747651 29808 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:56:30.748090 29808 webserver.cc:460] Webserver started at http://127.14.14.129:42429/ using document root <none> and password file <none>
I20260928 22:56:30.748330 29808 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:30.748400 29808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:30.748559 29808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:56:30.749818 29808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "d35c93b8f2e24aa7b7cf3a988167eca1"
format_stamp: "Formatted at 2026-09-28 22:56:30 on dist-test-slave-kzfr"
server_key: "07b6fc7c7268a1672cedb8b927b444b4"
server_key_iv: "b78ade7d3a2bccda0d52574a2ee4a63e"
server_key_version: "encryptionkey@0"
I20260928 22:56:30.750193 29808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "d35c93b8f2e24aa7b7cf3a988167eca1"
format_stamp: "Formatted at 2026-09-28 22:56:30 on dist-test-slave-kzfr"
server_key: "07b6fc7c7268a1672cedb8b927b444b4"
server_key_iv: "b78ade7d3a2bccda0d52574a2ee4a63e"
server_key_version: "encryptionkey@0"
I20260928 22:56:30.751607 29808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:30.752496 29823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:30.752686 29808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:30.752784 29808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "d35c93b8f2e24aa7b7cf3a988167eca1"
format_stamp: "Formatted at 2026-09-28 22:56:30 on dist-test-slave-kzfr"
server_key: "07b6fc7c7268a1672cedb8b927b444b4"
server_key_iv: "b78ade7d3a2bccda0d52574a2ee4a63e"
server_key_version: "encryptionkey@0"
I20260928 22:56:30.752868 29808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:30.794340 29808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:30.796627 29808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:30.796803 29808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:30.797053 29808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:30.797469 29808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:30.797536 29808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:30.797600 29808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:30.797631 29808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:30.803490 29808 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:43421
I20260928 22:56:30.803544 29936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:43421 every 8 connection(s)
I20260928 22:56:30.803879 29808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:56:30.806300 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 29808
I20260928 22:56:30.806411 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:56:30.806607 14394 external_mini_cluster.cc:1428] Setting key 2d9cd65658428b4d06c792930d9e6e9e
Sep 28 22:56:30 dist-test-slave-kzfr krb5kdc[29701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636190, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:30.815011 29786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:30.804662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:60341 (local address 127.14.14.190:33333)
0928 22:56:30.804760 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:30.804763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:30.804959 (+ 196us) server_negotiation.cc:407] Connection header received
0928 22:56:30.805437 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:30.805439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:30.805454 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:30.805518 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:30.806807 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:30.807484 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:30.809006 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:30.809373 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:30.809450 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:30.811985 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:30.811986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:30.811987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:30.811996 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:30.813438 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:30.813969 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:30.813970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:30.813970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:30.814026 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:30.814353 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:30.814354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:30.814354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:30.814470 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:30.814520 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:30.814867 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 22:56:30.814944 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 22:56:30.815287 29939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:30.804621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:60341)
0928 22:56:30.804805 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:30.804846 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:30.805305 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:30.805652 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:30.805656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:30.806336 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:30.806640 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:30.806647 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:30.807580 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:30.807581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:30.808909 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:30.808912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:30.809458 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:30.809458 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:30.809556 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:30.810129 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:30.810136 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:30.811889 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:30.813540 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:30.813548 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:30.813551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:30.813886 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:30.814094 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:30.814095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:30.814096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:30.814258 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:30.814540 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:30.814547 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:30.814722 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 22:56:30.814836 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 22:56:30.815013 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
I20260928 22:56:30.815874 29937 heartbeater.cc:347] Connected to a master server at 127.14.14.190:33333
I20260928 22:56:30.815976 29937 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:30.816169 29937 heartbeater.cc:511] Master 127.14.14.190:33333 requested a full tablet report, sending...
I20260928 22:56:30.816653 29746 ts_manager.cc:194] Registered new tserver with Master: d35c93b8f2e24aa7b7cf3a988167eca1 (127.14.14.129:43421)
I20260928 22:56:30.817384 29746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:60341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:56:30.837070 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:33333
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:30.919656 29942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:30.919929 29942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:30.919976 29942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:30.921289 29942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:30.921365 29942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:30.921411 29942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:56:30.922936 29942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:33333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:30.923203 29942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:30.923460 29942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:30.928634 29950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:30.928674 29947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:30.928844 29942 server_base.cc:1034] running on GCE node
W20260928 22:56:30.928642 29948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:30.929116 29942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:30.929358 29942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:30.930541 29942 hybrid_clock.cc:648] HybridClock initialized: now 1790636190930528 us; error 30 us; skew 500 ppm
Sep 28 22:56:30 dist-test-slave-kzfr krb5kdc[29701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636190, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:30.933020 29942 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:56:30.933493 29942 webserver.cc:460] Webserver started at http://127.14.14.130:42985/ using document root <none> and password file <none>
I20260928 22:56:30.933738 29942 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:30.933804 29942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:30.933938 29942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:56:30.934937 29942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "6b9ff8c1b7784202a31785075c39e82f"
format_stamp: "Formatted at 2026-09-28 22:56:30 on dist-test-slave-kzfr"
server_key: "0cec4eec5c4932482d37ec90733885f1"
server_key_iv: "53754e60c61208ce24785cf1d590163c"
server_key_version: "encryptionkey@0"
I20260928 22:56:30.935292 29942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "6b9ff8c1b7784202a31785075c39e82f"
format_stamp: "Formatted at 2026-09-28 22:56:30 on dist-test-slave-kzfr"
server_key: "0cec4eec5c4932482d37ec90733885f1"
server_key_iv: "53754e60c61208ce24785cf1d590163c"
server_key_version: "encryptionkey@0"
I20260928 22:56:30.936646 29942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:56:30.937390 29957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:30.937558 29942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:56:30.937683 29942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "6b9ff8c1b7784202a31785075c39e82f"
format_stamp: "Formatted at 2026-09-28 22:56:30 on dist-test-slave-kzfr"
server_key: "0cec4eec5c4932482d37ec90733885f1"
server_key_iv: "53754e60c61208ce24785cf1d590163c"
server_key_version: "encryptionkey@0"
I20260928 22:56:30.937762 29942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:30.948976 29942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:30.950984 29942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:30.951097 29942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:30.951267 29942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:30.951568 29942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:30.951601 29942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:30.951622 29942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:30.951668 29942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:30.957643 29942 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:44891
I20260928 22:56:30.957715 30070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:44891 every 8 connection(s)
I20260928 22:56:30.958040 29942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:56:30.961973 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 29942
I20260928 22:56:30.962054 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:56:30.962293 14394 external_mini_cluster.cc:1428] Setting key 26c664c676631862071dc6ba5912afdb
Sep 28 22:56:30 dist-test-slave-kzfr krb5kdc[29701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636190, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:30.968479 29786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:30.958893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:56019 (local address 127.14.14.190:33333)
0928 22:56:30.958954 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:30.958956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:30.959162 (+ 206us) server_negotiation.cc:407] Connection header received
0928 22:56:30.959710 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:30.959712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:30.959724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:30.959783 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:30.960717 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:30.961253 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:30.962577 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:30.962855 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:30.962927 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:30.965755 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:30.965756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:30.965756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:30.965764 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:30.967101 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:30.967508 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:30.967510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:30.967510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:30.967570 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:30.967836 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:30.967837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:30.967838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:30.967953 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:30.968028 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:30.968294 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 22:56:30.968384 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:56:30.968688 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:30.958861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:56019)
0928 22:56:30.959039 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:30.959082 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:30.959604 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:30.959852 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:30.959856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:30.960425 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:30.960617 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:30.960622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:30.961380 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:30.961381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:30.962459 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:30.962462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:30.962945 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:30.962946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:30.963103 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:30.963736 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:30.963747 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:30.965572 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:30.967196 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:30.967202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:30.967204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:30.967430 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:30.967637 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:30.967638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:30.967638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:30.967753 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:30.968018 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:30.968024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:30.968165 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 22:56:30.968275 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 22:56:30.968445 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
I20260928 22:56:30.969154 30071 heartbeater.cc:347] Connected to a master server at 127.14.14.190:33333
I20260928 22:56:30.969244 30071 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:30.969425 30071 heartbeater.cc:511] Master 127.14.14.190:33333 requested a full tablet report, sending...
I20260928 22:56:30.969836 29746 ts_manager.cc:194] Registered new tserver with Master: 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891)
I20260928 22:56:30.970208 29746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:56019
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:56:30.992563 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:33333
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:31.073891 30076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:31.074105 30076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:31.074131 30076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:31.075248 30076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:31.075289 30076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:31.075314 30076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:56:31.076721 30076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:33333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:31.076973 30076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:31.077171 30076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:31.082712 30081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:31.082712 30084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:31.082784 30082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:31.082902 30076 server_base.cc:1034] running on GCE node
I20260928 22:56:31.083213 30076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:31.083511 30076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:31.084728 30076 hybrid_clock.cc:648] HybridClock initialized: now 1790636191084707 us; error 28 us; skew 500 ppm
Sep 28 22:56:31 dist-test-slave-kzfr krb5kdc[29701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636191, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:31.087512 30076 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:56:31.088017 30076 webserver.cc:460] Webserver started at http://127.14.14.131:37755/ using document root <none> and password file <none>
I20260928 22:56:31.088230 30076 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:31.088279 30076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:31.088374 30076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:56:31.089407 30076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "95f2a34421ca4c5aa31aeddc782ca1a6"
format_stamp: "Formatted at 2026-09-28 22:56:31 on dist-test-slave-kzfr"
server_key: "b2bad76a58e6aa34a11f8a670413695c"
server_key_iv: "d57b5a41cd613f90f563de4cf5919681"
server_key_version: "encryptionkey@0"
I20260928 22:56:31.089829 30076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "95f2a34421ca4c5aa31aeddc782ca1a6"
format_stamp: "Formatted at 2026-09-28 22:56:31 on dist-test-slave-kzfr"
server_key: "b2bad76a58e6aa34a11f8a670413695c"
server_key_iv: "d57b5a41cd613f90f563de4cf5919681"
server_key_version: "encryptionkey@0"
I20260928 22:56:31.091182 30076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:56:31.092010 30091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:31.092193 30076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:56:31.092360 30076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "95f2a34421ca4c5aa31aeddc782ca1a6"
format_stamp: "Formatted at 2026-09-28 22:56:31 on dist-test-slave-kzfr"
server_key: "b2bad76a58e6aa34a11f8a670413695c"
server_key_iv: "d57b5a41cd613f90f563de4cf5919681"
server_key_version: "encryptionkey@0"
I20260928 22:56:31.092427 30076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:31.106302 30076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:31.108717 30076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:31.108897 30076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:31.109148 30076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:31.109472 30076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:31.109506 30076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:31.109529 30076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:31.109674 30076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:31.115824 30076 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:37421
I20260928 22:56:31.115911 30204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:37421 every 8 connection(s)
I20260928 22:56:31.116220 30076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:56:31.117568 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 30076
I20260928 22:56:31.117698 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:56:31.117924 14394 external_mini_cluster.cc:1428] Setting key 9890fd4072cc801e8b35a04d2e394376
Sep 28 22:56:31 dist-test-slave-kzfr krb5kdc[29701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636191, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:31.126276 29786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:31.117014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:50455 (local address 127.14.14.190:33333)
0928 22:56:31.117082 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:31.117085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:31.117283 (+ 198us) server_negotiation.cc:407] Connection header received
0928 22:56:31.118024 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:31.118025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:31.118039 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:31.118091 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:31.119311 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.119833 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.120881 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.121096 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.121185 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:31.123423 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:31.123425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:31.123425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:31.123432 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:31.124754 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:31.125214 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:31.125215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:31.125216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:31.125280 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:31.125536 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:31.125537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:31.125538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:31.125697 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:31.125813 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:31.126092 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 22:56:31.126180 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:56:31.126493 30207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:31.116999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:50455)
0928 22:56:31.117152 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:31.117188 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:31.117905 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:31.118281 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:31.118285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:31.118998 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:31.119198 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:31.119202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.119939 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.119940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:31.120783 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:31.120785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.121211 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.121212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:31.121345 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:31.121943 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:31.121951 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:31.123307 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:31.124873 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:31.124881 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:31.124883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:31.125114 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:31.125350 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:31.125351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:31.125351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:31.125458 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:31.125806 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:31.125813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:31.125949 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 22:56:31.126069 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 22:56:31.126226 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260928 22:56:31.126986 30205 heartbeater.cc:347] Connected to a master server at 127.14.14.190:33333
I20260928 22:56:31.127099 30205 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:31.127281 30205 heartbeater.cc:511] Master 127.14.14.190:33333 requested a full tablet report, sending...
I20260928 22:56:31.127688 29746 ts_manager.cc:194] Registered new tserver with Master: 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421)
I20260928 22:56:31.128073 29746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:50455
I20260928 22:56:31.129529 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:56:31.136454 29786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:31.130876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.14.14.190:33333)
0928 22:56:31.130962 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:31.130963 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:31.130995 (+ 32us) server_negotiation.cc:407] Connection header received
0928 22:56:31.131046 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:31.131047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:31.131054 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:31.131106 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:31.131603 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.132086 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.132797 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.132990 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.133046 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:31.134204 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:31.134205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:31.134205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:31.134211 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:31.135488 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:31.135792 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:31.135793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:31.135793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:31.135840 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:31.135997 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:31.135998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:31.135998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:31.136127 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:31.136199 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:31.136301 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 22:56:31.136355 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 22:56:31.138183 29746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:56:31.140085 29746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:56:31.147109 30219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:31.142672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.14.14.129:43421)
0928 22:56:31.143628 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:31.143633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:31.143642 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:56:31.143681 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:31.143682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:31.143696 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:31.143794 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:31.143902 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.144580 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.145958 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.146689 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.146767 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:31.146825 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:31.146973 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 22:56:31.147024 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":921,"thread_start_us":122,"threads_started":1}
I20260928 22:56:31.147151 30218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:31.142564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:44891 (local address 127.0.0.1:41028)
0928 22:56:31.142732 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:31.142777 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:31.142845 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:31.143161 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:31.143163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:31.143166 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:31.143275 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:31.143281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.144639 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.144641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:31.146062 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:31.146065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.146914 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.146915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:31.146972 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:31.146975 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:56:31.147025 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 22:56:31.147057 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":57,"thread_start_us":57,"threads_started":1}
I20260928 22:56:31.147261 30220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:31.142803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.14.14.130:44891)
0928 22:56:31.142974 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:31.142981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:31.142994 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:56:31.143035 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:31.143037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:31.143054 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:31.143141 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:31.143375 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.144529 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.146150 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.146827 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.146901 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:31.146981 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:31.147135 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 22:56:31.147181 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260928 22:56:31.147089 30216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:31.142573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:43421 (local address 127.0.0.1:42174)
0928 22:56:31.143086 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:31.143094 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:31.143129 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:31.143790 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:31.143791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:31.143793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:31.143826 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:31.143828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.144701 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.144702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:31.145872 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:31.145874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.146776 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.146777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:31.146833 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:31.146845 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 22:56:31.146945 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 22:56:31.147004 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":24,"thread_start_us":90,"threads_started":1}
I20260928 22:56:31.148381 30005 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3f0a1feea14248b7113694e1242ee3 (DEFAULT_TABLE table=test-table [id=e902e0ca64e540e5bce24080401e2dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:56:31.148428 29871 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3f0a1feea14248b7113694e1242ee3 (DEFAULT_TABLE table=test-table [id=e902e0ca64e540e5bce24080401e2dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:56:31.148748 29871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3f0a1feea14248b7113694e1242ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:31.148754 30005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3f0a1feea14248b7113694e1242ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:31.149804 30217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:31.142565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37421 (local address 127.0.0.1:35356)
0928 22:56:31.142972 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:31.142981 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:31.143020 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:31.143525 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:31.143526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:31.143527 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:31.143568 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:31.143570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.144800 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.144801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:31.146063 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:31.146065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.147818 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.147819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:31.147896 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:31.147900 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:56:31.149684 (+ 1784us) client_negotiation.cc:240] Negotiation successful
0928 22:56:31.149733 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":122,"thread_start_us":75,"threads_started":1}
I20260928 22:56:31.150046 30221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:31.142787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.14.14.131:37421)
0928 22:56:31.143349 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:31.143354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:31.143364 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:56:31.143399 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:31.143401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:31.143416 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:31.143494 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:31.143651 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.144703 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.147015 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.147733 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.147807 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:31.147872 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:31.149918 (+ 2046us) server_negotiation.cc:299] Negotiation successful
0928 22:56:31.149964 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":106,"threads_started":1}
I20260928 22:56:31.151816 30223 tablet_bootstrap.cc:492] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f: Bootstrap starting.
I20260928 22:56:31.151888 30222 tablet_bootstrap.cc:492] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: Bootstrap starting.
I20260928 22:56:31.152426 30223 tablet_bootstrap.cc:654] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:31.152338 30139 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3f0a1feea14248b7113694e1242ee3 (DEFAULT_TABLE table=test-table [id=e902e0ca64e540e5bce24080401e2dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:56:31.152585 30222 tablet_bootstrap.cc:654] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:31.152648 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3f0a1feea14248b7113694e1242ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:31.152969 30223 log.cc:824] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:31.153126 30222 log.cc:824] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:31.153687 30223 tablet_bootstrap.cc:492] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f: No bootstrap required, opened a new log
I20260928 22:56:31.153762 30223 ts_tablet_manager.cc:1397] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:56:31.153918 30222 tablet_bootstrap.cc:492] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: No bootstrap required, opened a new log
I20260928 22:56:31.153987 30222 ts_tablet_manager.cc:1397] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:56:31.155459 30223 raft_consensus.cc:348] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } }
I20260928 22:56:31.155625 30223 raft_consensus.cc:374] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:31.155686 30223 raft_consensus.cc:729] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9ff8c1b7784202a31785075c39e82f, State: Initialized, Role: FOLLOWER
I20260928 22:56:31.155820 30223 consensus_queue.cc:260] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } }
I20260928 22:56:31.156046 30226 tablet_bootstrap.cc:492] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6: Bootstrap starting.
I20260928 22:56:31.156203 30223 ts_tablet_manager.cc:1428] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:56:31.156322 30071 heartbeater.cc:503] Master 127.14.14.190:33333 was elected leader, sending a full tablet report...
I20260928 22:56:31.156229 30222 raft_consensus.cc:348] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } }
I20260928 22:56:31.156423 30222 raft_consensus.cc:374] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:31.156492 30222 raft_consensus.cc:729] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c93b8f2e24aa7b7cf3a988167eca1, State: Initialized, Role: FOLLOWER
I20260928 22:56:31.156608 30222 consensus_queue.cc:260] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } }
I20260928 22:56:31.156806 30226 tablet_bootstrap.cc:654] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:31.157017 29937 heartbeater.cc:503] Master 127.14.14.190:33333 was elected leader, sending a full tablet report...
I20260928 22:56:31.157295 30222 ts_tablet_manager.cc:1428] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:56:31.157390 30226 log.cc:824] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:31.158188 30226 tablet_bootstrap.cc:492] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6: No bootstrap required, opened a new log
I20260928 22:56:31.158253 30226 ts_tablet_manager.cc:1397] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:56:31.159693 30226 raft_consensus.cc:348] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } }
I20260928 22:56:31.159821 30226 raft_consensus.cc:374] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:31.159842 30226 raft_consensus.cc:729] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f2a34421ca4c5aa31aeddc782ca1a6, State: Initialized, Role: FOLLOWER
I20260928 22:56:31.159910 30226 consensus_queue.cc:260] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } }
I20260928 22:56:31.160094 30226 ts_tablet_manager.cc:1428] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:56:31.160144 30205 heartbeater.cc:503] Master 127.14.14.190:33333 was elected leader, sending a full tablet report...
W20260928 22:56:31.208616 30072 tablet.cc:2369] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:56:31.304570 29938 tablet.cc:2369] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:56:31.366816 30206 tablet.cc:2369] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:56:31.388545 30228 raft_consensus.cc:482] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:56:31.388677 30228 raft_consensus.cc:504] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } }
I20260928 22:56:31.389027 30228 leader_election.cc:288] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421), 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891)
I20260928 22:56:31.392110 30220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:31.389346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:54533 (local address 127.14.14.130:44891)
0928 22:56:31.389416 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:31.389418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:31.389500 (+ 82us) server_negotiation.cc:407] Connection header received
0928 22:56:31.389558 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:31.389559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:31.389569 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:31.389673 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:31.389905 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.390358 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.391359 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.391737 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.391793 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:31.391801 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:31.391991 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 22:56:31.392032 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:56:31.392135 30221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:31.389364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:47705 (local address 127.14.14.131:37421)
0928 22:56:31.389419 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:31.389421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:31.389484 (+ 63us) server_negotiation.cc:407] Connection header received
0928 22:56:31.389571 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:31.389572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:31.389608 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:31.389675 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:31.389847 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.390344 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.391379 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.391750 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.391840 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:31.391848 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:31.392030 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 22:56:31.392069 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:56:31.392153 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:31.389216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37421 (local address 127.14.14.129:47705)
0928 22:56:31.389413 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:31.389424 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:31.389498 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:31.389663 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:31.389664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:31.389666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:31.389763 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:31.389765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.390436 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.390437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:31.391260 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:31.391262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.391862 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.391863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:31.391932 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:31.391935 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:56:31.392009 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 22:56:31.392045 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":13,"thread_start_us":91,"threads_started":1}
I20260928 22:56:31.392151 30232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:31.389203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:44891 (local address 127.14.14.129:54533)
0928 22:56:31.389404 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:31.389416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:31.389494 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:31.389704 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:31.389705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:31.389708 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:31.389793 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:31.389816 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.390479 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.390480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:31.391257 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:31.391259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.391834 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.391835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:31.391907 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:31.391911 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:56:31.391963 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 22:56:31.392024 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":110,"threads_started":1}
I20260928 22:56:31.392612 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3f0a1feea14248b7113694e1242ee3" candidate_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9ff8c1b7784202a31785075c39e82f" is_pre_election: true
I20260928 22:56:31.392724 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3f0a1feea14248b7113694e1242ee3" candidate_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" is_pre_election: true
I20260928 22:56:31.392793 30025 raft_consensus.cc:2454] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c93b8f2e24aa7b7cf3a988167eca1 in term 0.
I20260928 22:56:31.392854 30159 raft_consensus.cc:2454] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c93b8f2e24aa7b7cf3a988167eca1 in term 0.
I20260928 22:56:31.393023 29825 leader_election.cc:302] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b9ff8c1b7784202a31785075c39e82f, d35c93b8f2e24aa7b7cf3a988167eca1; no voters:
I20260928 22:56:31.393157 30228 raft_consensus.cc:2781] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:56:31.393213 30228 raft_consensus.cc:482] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:56:31.393240 30228 raft_consensus.cc:3037] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:31.394027 30228 raft_consensus.cc:504] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } }
I20260928 22:56:31.394178 30228 leader_election.cc:288] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 1 election: Requested vote from peers 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421), 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891)
I20260928 22:56:31.394405 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3f0a1feea14248b7113694e1242ee3" candidate_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6"
I20260928 22:56:31.394405 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3f0a1feea14248b7113694e1242ee3" candidate_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9ff8c1b7784202a31785075c39e82f"
I20260928 22:56:31.394490 30025 raft_consensus.cc:3037] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:31.394490 30159 raft_consensus.cc:3037] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:31.395393 30159 raft_consensus.cc:2454] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c93b8f2e24aa7b7cf3a988167eca1 in term 1.
I20260928 22:56:31.395403 30025 raft_consensus.cc:2454] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c93b8f2e24aa7b7cf3a988167eca1 in term 1.
I20260928 22:56:31.395548 29826 leader_election.cc:302] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95f2a34421ca4c5aa31aeddc782ca1a6, d35c93b8f2e24aa7b7cf3a988167eca1; no voters:
I20260928 22:56:31.395637 30228 raft_consensus.cc:2781] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:56:31.395848 30228 raft_consensus.cc:686] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 LEADER]: Becoming Leader. State: Replica: d35c93b8f2e24aa7b7cf3a988167eca1, State: Running, Role: LEADER
I20260928 22:56:31.395967 30228 consensus_queue.cc:237] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } }
I20260928 22:56:31.396842 29746 catalog_manager.cc:5495] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c93b8f2e24aa7b7cf3a988167eca1 (127.14.14.129). New cstate: current_term: 1 leader_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } health_report { overall_health: UNKNOWN } } }
I20260928 22:56:31.428689 30219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:31.426282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42176 (local address 127.14.14.129:43421)
0928 22:56:31.426378 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:31.426380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:31.426389 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:56:31.426437 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:31.426438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:31.426455 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:31.426535 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:56:31.426672 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.427144 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.427913 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:31.428106 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:31.428180 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:31.428355 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:56:31.428459 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:56:31.428556 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:31.428574 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 22:56:31.428593 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 22:56:31.430622 30159 raft_consensus.cc:1261] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 1 FOLLOWER]: Refusing update from remote peer d35c93b8f2e24aa7b7cf3a988167eca1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:56:31.430624 30025 raft_consensus.cc:1261] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f [term 1 FOLLOWER]: Refusing update from remote peer d35c93b8f2e24aa7b7cf3a988167eca1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:56:31.430979 30228 consensus_queue.cc:1035] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:56:31.431141 30233 consensus_queue.cc:1035] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:56:31.433895 30238 mvcc.cc:205] Tried to move back new op lower bound from 7334445840096702464 to 7334445839958556672. Current Snapshot: MvccSnapshot[applied={T|T < 7334445840096702464}]
I20260928 22:56:31.435035 30237 mvcc.cc:205] Tried to move back new op lower bound from 7334445840096702464 to 7334445839958556672. Current Snapshot: MvccSnapshot[applied={T|T < 7334445840096702464}]
I20260928 22:56:31.435766 29745 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:56:31.435866 29745 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:56:31.436636 29745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Sending DeleteTablet for 3 replicas of tablet 8d3f0a1feea14248b7113694e1242ee3
I20260928 22:56:31.436964 29871 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3f0a1feea14248b7113694e1242ee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:31 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:42174
I20260928 22:56:31.437002 30139 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3f0a1feea14248b7113694e1242ee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:31 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:35356
I20260928 22:56:31.437170 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 29808
I20260928 22:56:31.437198 30247 tablet_replica.cc:330] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6: stopping tablet replica
I20260928 22:56:31.437963 30247 raft_consensus.cc:2229] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:56:31.438037 30247 raft_consensus.cc:2258] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:56:31.438426 30247 ts_tablet_manager.cc:1905] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:56:31.439517 30005 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3f0a1feea14248b7113694e1242ee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:31 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:41028
I20260928 22:56:31.439688 30248 tablet_replica.cc:330] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f: stopping tablet replica
I20260928 22:56:31.439780 30248 raft_consensus.cc:2229] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:56:31.439880 30248 raft_consensus.cc:2258] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:56:31.440279 30248 ts_tablet_manager.cc:1905] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:56:31.441548 30247 ts_tablet_manager.cc:1918] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:56:31.441664 30247 log.cc:1196] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/8d3f0a1feea14248b7113694e1242ee3
I20260928 22:56:31.441887 30248 ts_tablet_manager.cc:1918] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:56:31.441962 30248 log.cc:1196] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/8d3f0a1feea14248b7113694e1242ee3
I20260928 22:56:31.442011 30247 ts_tablet_manager.cc:1939] T 8d3f0a1feea14248b7113694e1242ee3 P 95f2a34421ca4c5aa31aeddc782ca1a6: Deleting consensus metadata
I20260928 22:56:31.442229 30248 ts_tablet_manager.cc:1939] T 8d3f0a1feea14248b7113694e1242ee3 P 6b9ff8c1b7784202a31785075c39e82f: Deleting consensus metadata
I20260928 22:56:31.442430 29732 catalog_manager.cc:4841] TS 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421): tablet 8d3f0a1feea14248b7113694e1242ee3 (table test-table [id=e902e0ca64e540e5bce24080401e2dfe]) successfully deleted
I20260928 22:56:31.442600 29731 catalog_manager.cc:4841] TS 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891): tablet 8d3f0a1feea14248b7113694e1242ee3 (table test-table [id=e902e0ca64e540e5bce24080401e2dfe]) successfully deleted
W20260928 22:56:31.443454 29733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:43421 (error 108)
I20260928 22:56:31.443652 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 29942
I20260928 22:56:31.444331 30217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:31.443845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:43421 (local address 127.0.0.1:42182)
0928 22:56:31.444267 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:31.444305 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402}
W20260928 22:56:31.444486 29733 catalog_manager.cc:4584] TS d35c93b8f2e24aa7b7cf3a988167eca1 (127.14.14.129:43421): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d3f0a1feea14248b7113694e1242ee3: Network error: Client connection negotiation failed: client connection to 127.14.14.129:43421: connect: Connection refused (error 111)
I20260928 22:56:31.448728 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 30076
I20260928 22:56:31.453320 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 29714
I20260928 22:56:51.458693 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:33333
--webserver_interface=127.14.14.190
--webserver_port=44091
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:33333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:56:51.544818 30250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:51.545019 30250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:51.545040 30250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:51.546265 30250 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 22:56:51.546306 30250 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 22:56:51.546319 30250 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 22:56:51.546331 30250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:56:51.546344 30250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:51.546381 30250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:51.546408 30250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:56:51.546425 30250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:56:51.547884 30250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:33333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:33333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=44091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:51.548141 30250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:51.548373 30250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:51.557255 30256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:51.557415 30259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:51.557488 30257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:51.557531 30250 server_base.cc:1034] running on GCE node
I20260928 22:56:51.557860 30250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:51.558197 30250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:51.559381 30250 hybrid_clock.cc:648] HybridClock initialized: now 1790636211559336 us; error 48 us; skew 500 ppm
Sep 28 22:56:51 dist-test-slave-kzfr krb5kdc[29701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636211, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:51.562139 30250 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:56:51.562709 30250 webserver.cc:460] Webserver started at http://127.14.14.190:44091/ using document root <none> and password file <none>
I20260928 22:56:51.562975 30250 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:51.563033 30250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:51.564530 30250 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:51.565217 30266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:51.565510 30250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:51.565630 30250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "24fd1c1861fc4832a7c8cb28644b36b4"
format_stamp: "Formatted at 2026-09-28 22:56:30 on dist-test-slave-kzfr"
server_key: "75004178183ce921b2cdf59a201d5599"
server_key_iv: "b50f83b0745618f2974c78bcf8e2f21c"
server_key_version: "encryptionkey@0"
I20260928 22:56:51.566043 30250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:51.574576 30250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:51.576885 30250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:51.577039 30250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:51.581192 30250 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:33333
I20260928 22:56:51.581382 30318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:33333 every 8 connection(s)
I20260928 22:56:51.581708 30250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:56:51.583298 30319 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:56:51.583899 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 30250
I20260928 22:56:51.583969 30319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Bootstrap starting.
I20260928 22:56:51.584218 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:43421
--local_ip_for_outbound_sockets=127.14.14.129
--tserver_master_addrs=127.14.14.190:33333
--webserver_port=42429
--webserver_interface=127.14.14.129
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:56:51.593070 30319 log.cc:824] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:51.595291 30319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:56:51.595645 30319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Bootstrap complete.
I20260928 22:56:51.598811 30319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } }
I20260928 22:56:51.599129 30319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd1c1861fc4832a7c8cb28644b36b4, State: Initialized, Role: FOLLOWER
I20260928 22:56:51.599315 30319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } }
I20260928 22:56:51.599400 30319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:56:51.599464 30319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:56:51.599520 30319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:56:51.600734 30319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } }
I20260928 22:56:51.600872 30319 leader_election.cc:302] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24fd1c1861fc4832a7c8cb28644b36b4; no voters:
I20260928 22:56:51.601081 30319 leader_election.cc:288] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 22:56:51.601142 30323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:56:51.601339 30323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 2 LEADER]: Becoming Leader. State: Replica: 24fd1c1861fc4832a7c8cb28644b36b4, State: Running, Role: LEADER
I20260928 22:56:51.601400 30319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:56:51.601486 30323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } }
I20260928 22:56:51.601800 30324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } } }
I20260928 22:56:51.601912 30324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:51.601799 30323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24fd1c1861fc4832a7c8cb28644b36b4. Latest consensus state: current_term: 2 leader_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } } }
I20260928 22:56:51.602129 30323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [sys.catalog]: This master's current role is: LEADER
I20260928 22:56:51.602113 30329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:56:51.603025 30329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:56:51.603179 30329 catalog_manager.cc:1259] Loaded cluster ID: 77119e044c8448a6a17453431774715c
I20260928 22:56:51.603440 30329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:56:51.604607 30329 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:56:51.604779 30329 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Loaded TSK: 0
I20260928 22:56:51.618829 30329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Generated new TSK 1
I20260928 22:56:51.618947 30329 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:56:51.672902 30321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:51.673115 30321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:51.673141 30321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:51.674392 30321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:51.674437 30321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:51.674464 30321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:56:51.675921 30321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:43421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=42429
--tserver_master_addrs=127.14.14.190:33333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:51.676167 30321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:51.676402 30321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:51.682778 30347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:51.682915 30349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:51.682998 30346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:51.683205 30321 server_base.cc:1034] running on GCE node
I20260928 22:56:51.683499 30321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:51.683786 30321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:51.684965 30321 hybrid_clock.cc:648] HybridClock initialized: now 1790636211684939 us; error 45 us; skew 500 ppm
Sep 28 22:56:51 dist-test-slave-kzfr krb5kdc[29701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636211, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:51.691483 30321 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:56:51.692061 30321 webserver.cc:460] Webserver started at http://127.14.14.129:42429/ using document root <none> and password file <none>
I20260928 22:56:51.692333 30321 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:51.692430 30321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:51.694098 30321 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:56:51.694963 30356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:51.695218 30321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:56:51.695327 30321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "d35c93b8f2e24aa7b7cf3a988167eca1"
format_stamp: "Formatted at 2026-09-28 22:56:30 on dist-test-slave-kzfr"
server_key: "07b6fc7c7268a1672cedb8b927b444b4"
server_key_iv: "b78ade7d3a2bccda0d52574a2ee4a63e"
server_key_version: "encryptionkey@0"
I20260928 22:56:51.695747 30321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:51.725104 30321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:51.727676 30321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:51.727893 30321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:51.728188 30321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:51.728703 30363 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:56:51.730954 30321 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:56:51.731036 30321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260928 22:56:51.731091 30321 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:56:51.731860 30321 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:56:51.731926 30321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 22:56:51.731974 30363 tablet_bootstrap.cc:492] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: Bootstrap starting.
I20260928 22:56:51.739876 30321 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:43421
I20260928 22:56:51.739928 30470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:43421 every 8 connection(s)
I20260928 22:56:51.740345 30321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:56:51.743367 30363 log.cc:824] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:51.746240 30363 tablet_bootstrap.cc:492] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:56:51.746634 30363 tablet_bootstrap.cc:492] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: Bootstrap complete.
I20260928 22:56:51.747967 30363 ts_tablet_manager.cc:1397] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
Sep 28 22:56:51 dist-test-slave-kzfr krb5kdc[29701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636211, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:51.749322 30363 raft_consensus.cc:348] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } }
I20260928 22:56:51.749756 30363 raft_consensus.cc:729] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c93b8f2e24aa7b7cf3a988167eca1, State: Initialized, Role: FOLLOWER
I20260928 22:56:51.750044 30363 consensus_queue.cc:260] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } }
I20260928 22:56:51.750133 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 30321
I20260928 22:56:51.750419 30363 ts_tablet_manager.cc:1428] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:56:51.750458 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:44891
--local_ip_for_outbound_sockets=127.14.14.130
--tserver_master_addrs=127.14.14.190:33333
--webserver_port=42985
--webserver_interface=127.14.14.130
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:56:51.753456 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:51.741098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:51909)
0928 22:56:51.741409 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:51.741451 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:51.742106 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:51.742601 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:51.742605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:51.742899 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:51.743098 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:51.743102 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:51.744652 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:51.744653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:51.745533 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:51.745535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:51.746270 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:51.746272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:51.746431 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:51.747043 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:51.747052 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:51.748624 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:51.751230 (+ 2606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:51.751237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:51.751240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:51.751657 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:51.752009 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:51.752010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:51.752011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:51.752149 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:51.752643 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:51.752652 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:51.752850 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 22:56:51.752987 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 22:56:51.753189 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":93,"threads_started":1}
I20260928 22:56:51.753620 30474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:51.741187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:51909 (local address 127.14.14.190:33333)
0928 22:56:51.741868 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:51.741877 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:51.741923 (+ 46us) server_negotiation.cc:407] Connection header received
0928 22:56:51.742358 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:51.742364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:51.742409 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:51.742568 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:51.743226 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:51.744496 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:51.745695 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:51.746147 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:51.746281 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:51.748743 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:51.748750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:51.748750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:51.748767 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:51.751112 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:51.751782 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:51.751786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:51.751787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:51.751917 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:51.752231 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:51.752232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:51.752232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:51.752544 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:51.752644 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:51.753071 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 22:56:51.753388 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":105,"threads_started":1}
I20260928 22:56:51.754724 30471 heartbeater.cc:347] Connected to a master server at 127.14.14.190:33333
I20260928 22:56:51.754851 30471 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:51.755066 30471 heartbeater.cc:511] Master 127.14.14.190:33333 requested a full tablet report, sending...
I20260928 22:56:51.755908 30283 ts_manager.cc:194] Registered new tserver with Master: d35c93b8f2e24aa7b7cf3a988167eca1 (127.14.14.129:43421)
I20260928 22:56:51.757207 30283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:51909
I20260928 22:56:51.758428 30471 heartbeater.cc:503] Master 127.14.14.190:33333 was elected leader, sending a full tablet report...
I20260928 22:56:51.765802 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:51.756618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:43421 (local address 127.0.0.1:40658)
0928 22:56:51.761743 (+ 5125us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:51.761798 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:51.761910 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:51.762196 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:51.762199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:51.762201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:51.762277 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:51.762280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:51.763237 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:51.763238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:51.764158 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:51.764161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:51.765368 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:51.765369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:51.765490 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:51.765504 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:56:51.765658 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 22:56:51.765732 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5081,"thread_start_us":97,"threads_started":1}
I20260928 22:56:51.765817 30479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:51.756908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.14.14.129:43421)
0928 22:56:51.757085 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:51.757091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:51.761901 (+ 4810us) server_negotiation.cc:407] Connection header received
0928 22:56:51.762065 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:51.762068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:51.762085 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:51.762189 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:51.762357 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:51.763131 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:51.764267 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:51.765110 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:51.765210 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:51.765306 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:51.765682 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 22:56:51.765734 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":111,"threads_started":1}
I20260928 22:56:51.766362 30405 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3f0a1feea14248b7113694e1242ee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:31 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:40658
I20260928 22:56:51.768287 30480 tablet_replica.cc:330] stopping tablet replica
I20260928 22:56:51.768415 30480 raft_consensus.cc:2229] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:56:51.768481 30480 raft_consensus.cc:2258] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:56:51.768857 30480 ts_tablet_manager.cc:1905] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:56:51.771647 30480 ts_tablet_manager.cc:1918] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:56:51.771740 30480 log.cc:1196] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal/wals/8d3f0a1feea14248b7113694e1242ee3
I20260928 22:56:51.772007 30480 ts_tablet_manager.cc:1939] T 8d3f0a1feea14248b7113694e1242ee3 P d35c93b8f2e24aa7b7cf3a988167eca1: Deleting consensus metadata
I20260928 22:56:51.772611 30270 catalog_manager.cc:4841] TS d35c93b8f2e24aa7b7cf3a988167eca1 (127.14.14.129:43421): tablet 8d3f0a1feea14248b7113694e1242ee3 (table test-table [id=e902e0ca64e540e5bce24080401e2dfe]) successfully deleted
W20260928 22:56:51.845356 30477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:51.845553 30477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:51.845621 30477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:51.846843 30477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:51.846889 30477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:51.846915 30477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:56:51.848340 30477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:44891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=42985
--tserver_master_addrs=127.14.14.190:33333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:51.848573 30477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:51.848799 30477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:51.853866 30485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:51.853921 30486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:51.853976 30477 server_base.cc:1034] running on GCE node
W20260928 22:56:51.853866 30488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:51.854354 30477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:51.854601 30477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:51.855756 30477 hybrid_clock.cc:648] HybridClock initialized: now 1790636211855749 us; error 38 us; skew 500 ppm
Sep 28 22:56:51 dist-test-slave-kzfr krb5kdc[29701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636211, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:51.858335 30477 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:56:51.858779 30477 webserver.cc:460] Webserver started at http://127.14.14.130:42985/ using document root <none> and password file <none>
I20260928 22:56:51.859020 30477 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:51.859077 30477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:51.860525 30477 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:56:51.861202 30495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:51.861446 30477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:56:51.861511 30477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "6b9ff8c1b7784202a31785075c39e82f"
format_stamp: "Formatted at 2026-09-28 22:56:30 on dist-test-slave-kzfr"
server_key: "0cec4eec5c4932482d37ec90733885f1"
server_key_iv: "53754e60c61208ce24785cf1d590163c"
server_key_version: "encryptionkey@0"
I20260928 22:56:51.861964 30477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:51.878518 30477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:51.880364 30477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:51.880482 30477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:51.880671 30477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:51.880986 30477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:51.881019 30477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:51.881042 30477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:51.881090 30477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:51.886837 30477 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:44891
I20260928 22:56:51.886931 30608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:44891 every 8 connection(s)
I20260928 22:56:51.887255 30477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:56:51 dist-test-slave-kzfr krb5kdc[29701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636211, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:51.895939 30474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:51.887998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:40367 (local address 127.14.14.190:33333)
0928 22:56:51.888075 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:51.888078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:51.888266 (+ 188us) server_negotiation.cc:407] Connection header received
0928 22:56:51.888784 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:51.888785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:51.888798 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:51.888850 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:51.889803 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:51.890289 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:51.891233 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:51.891461 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:51.891512 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:51.893527 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:51.893529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:51.893529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:51.893536 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:51.894779 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:51.895139 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:51.895140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:51.895140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:51.895189 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:51.895392 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:51.895392 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:51.895393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:51.895533 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:51.895586 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:51.895801 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 22:56:51.895870 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 22:56:51.896175 30611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:51.887976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:40367)
0928 22:56:51.888150 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:51.888183 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:51.888657 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:51.888913 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:51.888916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:51.889464 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:51.889726 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:51.889731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:51.890363 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:51.890364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:51.891160 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:51.891162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:51.891530 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:51.891530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:51.891627 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:51.892036 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:51.892044 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:51.893436 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:51.894862 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:51.894868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:51.894869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:51.895076 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:51.895244 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:51.895245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:51.895245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:51.895338 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:51.895586 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:51.895590 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:51.895696 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 22:56:51.895774 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:56:51.895914 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
I20260928 22:56:51.896715 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 30477
I20260928 22:56:51.896869 30609 heartbeater.cc:347] Connected to a master server at 127.14.14.190:33333
I20260928 22:56:51.896994 30609 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:51.897040 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:37421
--local_ip_for_outbound_sockets=127.14.14.131
--tserver_master_addrs=127.14.14.190:33333
--webserver_port=37755
--webserver_interface=127.14.14.131
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:56:51.897204 30609 heartbeater.cc:511] Master 127.14.14.190:33333 requested a full tablet report, sending...
I20260928 22:56:51.897701 30283 ts_manager.cc:194] Registered new tserver with Master: 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891)
I20260928 22:56:51.898105 30283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:40367
W20260928 22:56:51.977979 30612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:56:51.978209 30612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:56:51.978245 30612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:56:51.979377 30612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:56:51.979444 30612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:56:51.979491 30612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:56:51.980940 30612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:37421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=37755
--tserver_master_addrs=127.14.14.190:33333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:56:51.981210 30612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:56:51.981455 30612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:56:51.987813 30620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:51.987823 30618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:56:51.987823 30617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:56:51.988072 30612 server_base.cc:1034] running on GCE node
I20260928 22:56:51.988302 30612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:56:51.988556 30612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:56:51.989715 30612 hybrid_clock.cc:648] HybridClock initialized: now 1790636211989712 us; error 37 us; skew 500 ppm
Sep 28 22:56:51 dist-test-slave-kzfr krb5kdc[29701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636211, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:56:51.992280 30612 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:56:51.992744 30612 webserver.cc:460] Webserver started at http://127.14.14.131:37755/ using document root <none> and password file <none>
I20260928 22:56:51.993007 30612 fs_manager.cc:359] Metadata directory not provided
I20260928 22:56:51.993093 30612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:56:51.994554 30612 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:51.995265 30627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:51.995601 30612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:51.995705 30612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "95f2a34421ca4c5aa31aeddc782ca1a6"
format_stamp: "Formatted at 2026-09-28 22:56:31 on dist-test-slave-kzfr"
server_key: "b2bad76a58e6aa34a11f8a670413695c"
server_key_iv: "d57b5a41cd613f90f563de4cf5919681"
server_key_version: "encryptionkey@0"
I20260928 22:56:51.996102 30612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:56:52.016538 30612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:56:52.018724 30612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:56:52.018905 30612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:56:52.019156 30612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:56:52.019474 30612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:56:52.019531 30612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:52.019573 30612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:56:52.019618 30612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:52.025648 30612 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:37421
I20260928 22:56:52.025668 30740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:37421 every 8 connection(s)
I20260928 22:56:52.026060 30612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:56:52.032438 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 30612
Sep 28 22:56:52 dist-test-slave-kzfr krb5kdc[29701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636211, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:56:52.036360 30474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.026815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:60253 (local address 127.14.14.190:33333)
0928 22:56:52.026891 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:52.026893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:52.027067 (+ 174us) server_negotiation.cc:407] Connection header received
0928 22:56:52.027682 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:52.027684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:52.027697 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:52.027771 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:52.028747 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.029244 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.030391 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.030614 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.030698 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:52.033162 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:52.033163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:52.033164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:52.033171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:52.034870 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:52.035360 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:52.035361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:52.035361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:52.035420 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:52.035693 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:52.035693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:52.035694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:52.035787 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:52.035836 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:52.036155 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 22:56:52.036237 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 22:56:52.036438 30743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.026732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:60253)
0928 22:56:52.026909 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:52.026960 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:52.027560 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:52.027851 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:52.027855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:52.028435 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:56:52.028627 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:52.028632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.029340 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.029341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:52.030299 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:52.030301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.030708 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.030709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:52.030834 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:52.031343 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:56:52.031351 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:56:52.033070 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:56:52.034962 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:52.034970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:52.034972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:52.035271 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:52.035489 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:56:52.035489 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:56:52.035490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:56:52.035626 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:56:52.035848 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:56:52.035855 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:56:52.036012 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 22:56:52.036119 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 22:56:52.036239 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260928 22:56:52.037187 30741 heartbeater.cc:347] Connected to a master server at 127.14.14.190:33333
I20260928 22:56:52.037297 30741 heartbeater.cc:464] Registering TS with master...
I20260928 22:56:52.037508 30741 heartbeater.cc:511] Master 127.14.14.190:33333 requested a full tablet report, sending...
I20260928 22:56:52.037938 30283 ts_manager.cc:194] Registered new tserver with Master: 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421)
I20260928 22:56:52.038425 30283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:60253
I20260928 22:56:52.040390 30745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.032864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52052 (local address 127.14.14.190:33333)
0928 22:56:52.033195 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:52.033196 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:52.033203 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:56:52.033223 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:52.033224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:52.033233 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:52.033281 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:52.034026 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.034858 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.036021 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.036299 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.036364 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:52.037849 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:52.037851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:52.037851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:52.037858 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:52.039242 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:52.039632 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:52.039633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:52.039633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:52.039692 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:52.039889 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:52.039889 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:52.039889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:52.039981 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:52.040111 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:52.040143 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 22:56:52.040211 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":90,"threads_started":1}
I20260928 22:56:52.041117 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:56:52.043711 30745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.041434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52058 (local address 127.14.14.190:33333)
0928 22:56:52.041470 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:52.041471 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:52.041691 (+ 220us) server_negotiation.cc:407] Connection header received
0928 22:56:52.041732 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:52.041733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:52.041775 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:52.041841 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:56:52.041926 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.042358 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.043122 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.043312 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.043392 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:52.043514 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:56:52.043553 (+ 39us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 22:56:52.043667 (+ 114us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52058: token expired
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 22:56:52.043797 30745 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52058: token expired
W20260928 22:56:52.043891 30208 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.14.190:33333)
I20260928 22:56:52.049427 30745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.044165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52066 (local address 127.14.14.190:33333)
0928 22:56:52.044197 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:52.044199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:52.044204 (+ 5us) server_negotiation.cc:407] Connection header received
0928 22:56:52.044223 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:52.044223 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:52.044230 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:52.044273 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:56:52.044714 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.045153 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.045912 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.046092 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.046155 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:52.047202 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:56:52.047203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:56:52.047204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:56:52.047209 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:56:52.048413 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:52.048751 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:52.048752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:52.048752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:52.048802 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:56:52.048996 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:56:52.048997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:56:52.048997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:56:52.049118 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:56:52.049192 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:52.049298 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 22:56:52.049368 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 22:56:52.050644 30283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:56:52.052335 30283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:56:52.054766 30405 tablet_service.cc:1467] Processing CreateTablet for tablet 0a63809b68d44d5cbc7c89b004a7de08 (DEFAULT_TABLE table=test-table [id=b2cb3d20813a4cd2b45f1ebd148a24f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:56:52.055008 30405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a63809b68d44d5cbc7c89b004a7de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:52.056221 30363 tablet_bootstrap.cc:492] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1: Bootstrap starting.
I20260928 22:56:52.056578 30363 tablet_bootstrap.cc:654] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:52.057164 30363 tablet_bootstrap.cc:492] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1: No bootstrap required, opened a new log
I20260928 22:56:52.057219 30363 ts_tablet_manager.cc:1397] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:56:52.057336 30363 raft_consensus.cc:348] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:56:52.057407 30363 raft_consensus.cc:374] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:52.057457 30363 raft_consensus.cc:729] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c93b8f2e24aa7b7cf3a988167eca1, State: Initialized, Role: FOLLOWER
I20260928 22:56:52.057497 30363 consensus_queue.cc:260] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:56:52.057612 30363 ts_tablet_manager.cc:1428] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:56:52.058430 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.054701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.14.14.130:44891)
0928 22:56:52.054887 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:52.054892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:52.054971 (+ 79us) server_negotiation.cc:407] Connection header received
0928 22:56:52.055053 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:52.055055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:52.055071 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:52.055170 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:52.055268 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.055949 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.057251 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.058095 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.058173 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:52.058235 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:52.058313 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 22:56:52.058348 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":115,"threads_started":1}
I20260928 22:56:52.058419 30747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.054518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:44891 (local address 127.0.0.1:50434)
0928 22:56:52.054919 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:52.054927 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:52.054983 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:52.055165 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:52.055166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:52.055168 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:52.055218 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:52.055220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.056068 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.056069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:52.057149 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:52.057152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.058174 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.058175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:52.058235 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:52.058240 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:56:52.058299 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:56:52.058345 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":9,"thread_start_us":117,"threads_started":1}
I20260928 22:56:52.060256 30749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.056276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.14.14.131:37421)
0928 22:56:52.056565 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:52.056570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:52.056581 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:56:52.056641 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:52.056643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:52.056660 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:52.056741 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:52.056877 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.058063 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.059250 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.059926 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.059988 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:52.060024 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:52.060168 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 22:56:52.060196 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":95,"threads_started":1}
I20260928 22:56:52.060261 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.054448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37421 (local address 127.0.0.1:38176)
0928 22:56:52.054537 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:52.054549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:52.054584 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:52.056738 (+ 2154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:52.056738 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:52.056740 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:52.056784 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:52.056785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.058177 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.058178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:52.059173 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:52.059175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.060013 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.060014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:52.060086 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:52.060089 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:56:52.060151 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 22:56:52.060200 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 22:56:52.060169 30543 tablet_service.cc:1467] Processing CreateTablet for tablet 0a63809b68d44d5cbc7c89b004a7de08 (DEFAULT_TABLE table=test-table [id=b2cb3d20813a4cd2b45f1ebd148a24f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:56:52.060531 30543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a63809b68d44d5cbc7c89b004a7de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:52.061062 30675 tablet_service.cc:1467] Processing CreateTablet for tablet 0a63809b68d44d5cbc7c89b004a7de08 (DEFAULT_TABLE table=test-table [id=b2cb3d20813a4cd2b45f1ebd148a24f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:56:52.061426 30675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a63809b68d44d5cbc7c89b004a7de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:56:52.063618 30751 tablet_bootstrap.cc:492] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: Bootstrap starting.
I20260928 22:56:52.064199 30752 tablet_bootstrap.cc:492] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6: Bootstrap starting.
I20260928 22:56:52.064288 30751 tablet_bootstrap.cc:654] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:52.064829 30752 tablet_bootstrap.cc:654] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6: Neither blocks nor log segments found. Creating new log.
I20260928 22:56:52.064841 30751 log.cc:824] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:52.065464 30752 log.cc:824] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6: Log is configured to *not* fsync() on all Append() calls
I20260928 22:56:52.065721 30751 tablet_bootstrap.cc:492] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: No bootstrap required, opened a new log
I20260928 22:56:52.065797 30751 ts_tablet_manager.cc:1397] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:56:52.066498 30752 tablet_bootstrap.cc:492] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6: No bootstrap required, opened a new log
I20260928 22:56:52.066567 30752 ts_tablet_manager.cc:1397] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:56:52.067464 30751 raft_consensus.cc:348] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:56:52.067642 30751 raft_consensus.cc:374] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:52.067679 30751 raft_consensus.cc:729] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9ff8c1b7784202a31785075c39e82f, State: Initialized, Role: FOLLOWER
I20260928 22:56:52.067793 30751 consensus_queue.cc:260] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:56:52.068089 30751 ts_tablet_manager.cc:1428] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:56:52.068109 30609 heartbeater.cc:503] Master 127.14.14.190:33333 was elected leader, sending a full tablet report...
I20260928 22:56:52.068457 30752 raft_consensus.cc:348] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:56:52.068630 30752 raft_consensus.cc:374] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:56:52.068677 30752 raft_consensus.cc:729] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f2a34421ca4c5aa31aeddc782ca1a6, State: Initialized, Role: FOLLOWER
I20260928 22:56:52.068796 30752 consensus_queue.cc:260] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:56:52.069108 30752 ts_tablet_manager.cc:1428] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:56:52.069149 30741 heartbeater.cc:503] Master 127.14.14.190:33333 was elected leader, sending a full tablet report...
W20260928 22:56:52.137841 30610 tablet.cc:2369] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:56:52.234853 30476 raft_consensus.cc:482] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:56:52.235003 30476 raft_consensus.cc:504] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:56:52.235404 30476 leader_election.cc:288] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891), 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421)
I20260928 22:56:52.238585 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.235715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:38443 (local address 127.14.14.130:44891)
0928 22:56:52.235762 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:52.235764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:52.235776 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:56:52.235874 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:52.235875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:52.235887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:52.235982 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:52.236182 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.236843 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.237848 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.238238 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.238326 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:52.238335 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:52.238483 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 22:56:52.238520 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:56:52.238605 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.235631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:44891 (local address 127.14.14.129:38443)
0928 22:56:52.235708 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:52.235721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:52.235778 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:52.236015 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:52.236016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:52.236018 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:52.236096 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:52.236099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.236931 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.236932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:52.237754 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:52.237756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.238325 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.238326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:52.238405 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:52.238408 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:56:52.238476 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 22:56:52.238515 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":3}
I20260928 22:56:52.238870 30749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.235743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:38111 (local address 127.14.14.131:37421)
0928 22:56:52.235797 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:52.235799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:52.235840 (+ 41us) server_negotiation.cc:407] Connection header received
0928 22:56:52.235924 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:52.235925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:52.235934 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:52.235992 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:56:52.236684 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.237165 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.238161 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.238531 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.238616 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:52.238623 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:52.238752 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 22:56:52.238798 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:56:52.238890 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.235631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37421 (local address 127.14.14.129:38111)
0928 22:56:52.235783 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:52.235791 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:56:52.235846 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:56:52.236015 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:56:52.236016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:56:52.236018 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:56:52.236613 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:52.236617 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.237285 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.237287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:52.238058 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:56:52.238060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.238616 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.238617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:56:52.238683 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:52.238687 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:56:52.238741 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 22:56:52.238780 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":15,"thread_start_us":86,"threads_started":1}
I20260928 22:56:52.238977 30563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a63809b68d44d5cbc7c89b004a7de08" candidate_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9ff8c1b7784202a31785075c39e82f" is_pre_election: true
I20260928 22:56:52.239153 30563 raft_consensus.cc:2454] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c93b8f2e24aa7b7cf3a988167eca1 in term 0.
I20260928 22:56:52.239228 30695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a63809b68d44d5cbc7c89b004a7de08" candidate_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" is_pre_election: true
I20260928 22:56:52.239393 30695 raft_consensus.cc:2454] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c93b8f2e24aa7b7cf3a988167eca1 in term 0.
I20260928 22:56:52.239405 30358 leader_election.cc:302] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b9ff8c1b7784202a31785075c39e82f, d35c93b8f2e24aa7b7cf3a988167eca1; no voters:
I20260928 22:56:52.239530 30476 raft_consensus.cc:2781] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:56:52.239629 30476 raft_consensus.cc:482] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:56:52.239655 30476 raft_consensus.cc:3037] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:52.240401 30476 raft_consensus.cc:504] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:56:52.240540 30476 leader_election.cc:288] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 1 election: Requested vote from peers 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891), 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421)
I20260928 22:56:52.240706 30563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a63809b68d44d5cbc7c89b004a7de08" candidate_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9ff8c1b7784202a31785075c39e82f"
I20260928 22:56:52.240718 30695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a63809b68d44d5cbc7c89b004a7de08" candidate_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6"
I20260928 22:56:52.240787 30695 raft_consensus.cc:3037] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:52.240813 30563 raft_consensus.cc:3037] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:56:52.241474 30695 raft_consensus.cc:2454] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c93b8f2e24aa7b7cf3a988167eca1 in term 1.
I20260928 22:56:52.241616 30563 raft_consensus.cc:2454] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c93b8f2e24aa7b7cf3a988167eca1 in term 1.
I20260928 22:56:52.241724 30359 leader_election.cc:302] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95f2a34421ca4c5aa31aeddc782ca1a6, d35c93b8f2e24aa7b7cf3a988167eca1; no voters:
I20260928 22:56:52.241861 30476 raft_consensus.cc:2781] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:56:52.242035 30476 raft_consensus.cc:686] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 LEADER]: Becoming Leader. State: Replica: d35c93b8f2e24aa7b7cf3a988167eca1, State: Running, Role: LEADER
W20260928 22:56:52.242081 30472 tablet.cc:2369] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:56:52.242117 30476 consensus_queue.cc:237] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:56:52.242822 30283 catalog_manager.cc:5495] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c93b8f2e24aa7b7cf3a988167eca1 (127.14.14.129). New cstate: current_term: 1 leader_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } health_report { overall_health: HEALTHY } } }
W20260928 22:56:52.276600 30742 tablet.cc:2369] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:56:52.285531 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.283227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.14.14.129:43421)
0928 22:56:52.283360 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 22:56:52.283362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:56:52.283369 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:56:52.283408 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:56:52.283409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:56:52.283432 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:56:52.283504 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:56:52.283663 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.284155 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.284864 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:56:52.285068 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:56:52.285121 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:56:52.285250 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:56:52.285353 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:56:52.285409 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:56:52.285450 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 22:56:52.285468 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260928 22:56:52.287220 30563 raft_consensus.cc:1261] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f [term 1 FOLLOWER]: Refusing update from remote peer d35c93b8f2e24aa7b7cf3a988167eca1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:56:52.287215 30695 raft_consensus.cc:1261] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 1 FOLLOWER]: Refusing update from remote peer d35c93b8f2e24aa7b7cf3a988167eca1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:56:52.287635 30476 consensus_queue.cc:1035] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:56:52.287846 30758 consensus_queue.cc:1035] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:56:52.290105 30761 mvcc.cc:205] Tried to move back new op lower bound from 7334445925525872640 to 7334445925344276480. Current Snapshot: MvccSnapshot[applied={T|T < 7334445925525872640}]
I20260928 22:56:52.290278 30762 mvcc.cc:205] Tried to move back new op lower bound from 7334445925525872640 to 7334445925344276480. Current Snapshot: MvccSnapshot[applied={T|T < 7334445925525872640}]
I20260928 22:56:52.291214 30763 mvcc.cc:205] Tried to move back new op lower bound from 7334445925525872640 to 7334445925344276480. Current Snapshot: MvccSnapshot[applied={T|T < 7334445925525872640 or (T in {7334445925525872640})}]
I20260928 22:56:52.292059 30282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:56:52.292160 30282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:56:52.292920 30282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Sending DeleteTablet for 3 replicas of tablet 0a63809b68d44d5cbc7c89b004a7de08
I20260928 22:56:52.293294 30543 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a63809b68d44d5cbc7c89b004a7de08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:52 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:50434
I20260928 22:56:52.293552 30771 tablet_replica.cc:330] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: stopping tablet replica
I20260928 22:56:52.293679 30771 raft_consensus.cc:2229] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:56:52.293783 30771 raft_consensus.cc:2258] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:56:52.294266 30405 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a63809b68d44d5cbc7c89b004a7de08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:52 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:40658
I20260928 22:56:52.294339 30675 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a63809b68d44d5cbc7c89b004a7de08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:56:52 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:38176
I20260928 22:56:52.294512 30772 tablet_replica.cc:330] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1: stopping tablet replica
I20260928 22:56:52.294564 30773 tablet_replica.cc:330] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6: stopping tablet replica
I20260928 22:56:52.294579 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 30321
I20260928 22:56:52.294663 30772 raft_consensus.cc:2229] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:56:52.294770 30772 raft_consensus.cc:2258] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:56:52.294656 30773 raft_consensus.cc:2229] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:56:52.295410 30773 raft_consensus.cc:2258] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:56:52.295845 30773 ts_tablet_manager.cc:1905] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:56:52.297520 30773 ts_tablet_manager.cc:1918] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:56:52.297619 30773 log.cc:1196] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/0a63809b68d44d5cbc7c89b004a7de08
I20260928 22:56:52.297907 30773 ts_tablet_manager.cc:1939] T 0a63809b68d44d5cbc7c89b004a7de08 P 95f2a34421ca4c5aa31aeddc782ca1a6: Deleting consensus metadata
I20260928 22:56:52.298375 30269 catalog_manager.cc:4841] TS 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421): tablet 0a63809b68d44d5cbc7c89b004a7de08 (table test-table [id=b2cb3d20813a4cd2b45f1ebd148a24f7]) successfully deleted
I20260928 22:56:52.301811 30771 ts_tablet_manager.cc:1905] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 22:56:52.302999 30270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:43421 (error 108)
I20260928 22:56:52.303277 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 30477
I20260928 22:56:52.303467 30771 ts_tablet_manager.cc:1918] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:56:52.303532 30771 log.cc:1196] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/0a63809b68d44d5cbc7c89b004a7de08
I20260928 22:56:52.303954 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.303356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:43421 (local address 127.0.0.1:40670)
0928 22:56:52.303877 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:52.303927 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489}
W20260928 22:56:52.304073 30270 catalog_manager.cc:4584] TS d35c93b8f2e24aa7b7cf3a988167eca1 (127.14.14.129:43421): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a63809b68d44d5cbc7c89b004a7de08: Network error: Client connection negotiation failed: client connection to 127.14.14.129:43421: connect: Connection refused (error 111)
I20260928 22:56:52.308050 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 30612
I20260928 22:56:52.308085 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:56:52.307973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:44891 (local address 127.0.0.1:50450)
0928 22:56:52.308033 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 22:56:52.308064 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.130:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 22:56:52.313068 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 30250
I20260928 22:57:12.318677 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:33333
--webserver_interface=127.14.14.190
--webserver_port=44091
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:33333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:12.399907 30775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:12.400200 30775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:12.400238 30775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:12.401419 30775 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 22:57:12.401494 30775 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 22:57:12.401515 30775 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 22:57:12.401532 30775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:12.401567 30775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:12.401649 30775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:12.401679 30775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:12.401706 30775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:12.403290 30775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:33333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:33333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=44091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:12.403564 30775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:12.403795 30775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:57:12.409322 30775 server_base.cc:1034] running on GCE node
W20260928 22:57:12.409482 30783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:12.409495 30781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:12.409612 30780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:12.409909 30775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:12.410234 30775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:12.411406 30775 hybrid_clock.cc:648] HybridClock initialized: now 1790636232411375 us; error 49 us; skew 500 ppm
Sep 28 22:57:12 dist-test-slave-kzfr krb5kdc[29701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636232, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:12.418152 30775 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:57:12.418733 30775 webserver.cc:460] Webserver started at http://127.14.14.190:44091/ using document root <none> and password file <none>
I20260928 22:57:12.418994 30775 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:12.419071 30775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:12.420450 30775 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:12.421193 30790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:12.421478 30775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:12.421641 30775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "24fd1c1861fc4832a7c8cb28644b36b4"
format_stamp: "Formatted at 2026-09-28 22:56:30 on dist-test-slave-kzfr"
server_key: "75004178183ce921b2cdf59a201d5599"
server_key_iv: "b50f83b0745618f2974c78bcf8e2f21c"
server_key_version: "encryptionkey@0"
I20260928 22:57:12.422013 30775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:12.464686 30775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:12.466792 30775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:12.466977 30775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:12.470935 30775 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:33333
I20260928 22:57:12.470985 30842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:33333 every 8 connection(s)
I20260928 22:57:12.471382 30775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:12.472988 30843 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:57:12.473690 30843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Bootstrap starting.
I20260928 22:57:12.474301 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 30775
I20260928 22:57:12.474592 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:43421
--local_ip_for_outbound_sockets=127.14.14.129
--tserver_master_addrs=127.14.14.190:33333
--webserver_port=42429
--webserver_interface=127.14.14.129
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:57:12.482163 30843 log.cc:824] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:12.484627 30843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:57:12.484961 30843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Bootstrap complete.
I20260928 22:57:12.487257 30843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } }
I20260928 22:57:12.487564 30843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd1c1861fc4832a7c8cb28644b36b4, State: Initialized, Role: FOLLOWER
I20260928 22:57:12.487767 30843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } }
I20260928 22:57:12.487854 30843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:12.487910 30843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:12.487970 30843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 2 FOLLOWER]: Advancing to term 3
I20260928 22:57:12.489022 30843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } }
I20260928 22:57:12.489111 30843 leader_election.cc:302] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24fd1c1861fc4832a7c8cb28644b36b4; no voters:
I20260928 22:57:12.489307 30843 leader_election.cc:288] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 22:57:12.489373 30847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 22:57:12.489535 30847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [term 3 LEADER]: Becoming Leader. State: Replica: 24fd1c1861fc4832a7c8cb28644b36b4, State: Running, Role: LEADER
I20260928 22:57:12.489634 30843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:12.489666 30847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } }
I20260928 22:57:12.490151 30848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } } }
I20260928 22:57:12.490250 30848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:12.490475 30849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24fd1c1861fc4832a7c8cb28644b36b4. Latest consensus state: current_term: 3 leader_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fd1c1861fc4832a7c8cb28644b36b4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33333 } } }
I20260928 22:57:12.490592 30849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:12.490767 30856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:12.491649 30856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:12.491750 30856 catalog_manager.cc:1259] Loaded cluster ID: 77119e044c8448a6a17453431774715c
I20260928 22:57:12.491775 30856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:12.493027 30856 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:12.493221 30856 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Loaded TSK: 0
I20260928 22:57:12.493273 30856 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 24fd1c1861fc4832a7c8cb28644b36b4: Loaded TSK: 1
F20260928 22:57:12.493556 30856 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f744296b9ad google::LogMessage::Fail() at ??:0
@ 0x7f744296e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f744296b4de google::LogMessage::Flush() at ??:0
@ 0x7f744296ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f74450d8104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f74450d5d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f744a8c51cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f744a8c548e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f744a8c5594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f744a88bc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f744a895ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f74448390d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f744482e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f74456926db start_thread at ??:0
@ 0x7f744146961f clone at ??:0
W20260928 22:57:12.568269 30845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:12.568537 30845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:12.568575 30845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:12.569836 30845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:12.569893 30845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:12.569939 30845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:57:12.571414 30845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:43421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=42429
--tserver_master_addrs=127.14.14.190:33333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:12.571676 30845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:12.571899 30845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:12.577070 30887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:12.577046 30884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:12.577198 30845 server_base.cc:1034] running on GCE node
W20260928 22:57:12.577231 30885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:12.577463 30845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:12.577749 30845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:12.578912 30845 hybrid_clock.cc:648] HybridClock initialized: now 1790636232578898 us; error 31 us; skew 500 ppm
Sep 28 22:57:12 dist-test-slave-kzfr krb5kdc[29701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636232, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:12.581514 30845 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:57:12.582024 30845 webserver.cc:460] Webserver started at http://127.14.14.129:42429/ using document root <none> and password file <none>
I20260928 22:57:12.582258 30845 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:12.582324 30845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:12.583678 30845 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:12.584371 30894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:12.584528 30845 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:12.584610 30845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "d35c93b8f2e24aa7b7cf3a988167eca1"
format_stamp: "Formatted at 2026-09-28 22:56:30 on dist-test-slave-kzfr"
server_key: "07b6fc7c7268a1672cedb8b927b444b4"
server_key_iv: "b78ade7d3a2bccda0d52574a2ee4a63e"
server_key_version: "encryptionkey@0"
I20260928 22:57:12.584961 30845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:12.598165 30845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:12.600502 30845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:12.600612 30845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:12.600881 30845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:12.601366 30901 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:57:12.603417 30845 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:57:12.603488 30845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 22:57:12.603539 30845 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:57:12.604177 30845 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:57:12.604233 30845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 22:57:12.604314 30901 tablet_bootstrap.cc:492] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1: Bootstrap starting.
I20260928 22:57:12.611406 30845 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:43421
I20260928 22:57:12.611771 30845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:57:12.612298 31008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:43421 every 8 connection(s)
I20260928 22:57:12.615123 30901 log.cc:824] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:12.617218 30901 tablet_bootstrap.cc:492] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:57:12.617625 30901 tablet_bootstrap.cc:492] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1: Bootstrap complete.
I20260928 22:57:12.618557 30901 ts_tablet_manager.cc:1397] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20260928 22:57:12.619546 30901 raft_consensus.cc:348] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:57:12.619793 30901 raft_consensus.cc:729] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c93b8f2e24aa7b7cf3a988167eca1, State: Initialized, Role: FOLLOWER
I20260928 22:57:12.619834 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 30845
I20260928 22:57:12.619962 30901 consensus_queue.cc:260] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:57:12.620133 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:44891
--local_ip_for_outbound_sockets=127.14.14.130
--tserver_master_addrs=127.14.14.190:33333
--webserver_port=42985
--webserver_interface=127.14.14.130
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:57:12.620245 30901 ts_tablet_manager.cc:1428] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 22:57:12.698616 31011 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:57:12.612552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:50609)
0928 22:57:12.612729 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:12.612791 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:12.613557 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:12.698408 (+ 84851us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
W20260928 22:57:12.698894 31009 heartbeater.cc:644] Failed to heartbeat to 127.14.14.190:33333 (0 consecutive failures): Network error: Failed to ping master at 127.14.14.190:33333: Client connection negotiation failed: client connection to 127.14.14.190:33333: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 22:57:12.699312 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:12.699175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:36153)
0928 22:57:12.699232 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:12.699284 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 22:57:12.699708 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:12.699609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:58713)
0928 22:57:12.699659 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:12.699688 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 22:57:12.699839 31009 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:57:12.702425 31014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:12.702615 31014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:12.702662 31014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:12.703820 31014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:12.703859 31014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:12.703914 31014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:57:12.705386 31014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:44891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=42985
--tserver_master_addrs=127.14.14.190:33333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:12.705652 31014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:12.705897 31014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:12.710819 31022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:12.710898 31014 server_base.cc:1034] running on GCE node
W20260928 22:57:12.710846 31020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:12.711002 31019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:12.711257 31014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:12.711534 31014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:12.712706 31014 hybrid_clock.cc:648] HybridClock initialized: now 1790636232712694 us; error 45 us; skew 500 ppm
Sep 28 22:57:12 dist-test-slave-kzfr krb5kdc[29701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636232, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:12.715106 31014 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:57:12.715605 31014 webserver.cc:460] Webserver started at http://127.14.14.130:42985/ using document root <none> and password file <none>
I20260928 22:57:12.715831 31014 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:12.715900 31014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:12.717185 31014 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:12.717909 31029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:12.718066 31014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:57:12.718163 31014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "6b9ff8c1b7784202a31785075c39e82f"
format_stamp: "Formatted at 2026-09-28 22:56:30 on dist-test-slave-kzfr"
server_key: "0cec4eec5c4932482d37ec90733885f1"
server_key_iv: "53754e60c61208ce24785cf1d590163c"
server_key_version: "encryptionkey@0"
I20260928 22:57:12.718535 31014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:12.728330 31014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:12.730228 31014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:12.730381 31014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:12.730604 31014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:12.731021 31036 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 22:57:12.732432 31036 ts_tablet_manager.cc:1852] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 22:57:12.732702 31036 ts_tablet_manager.cc:1905] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:57:12.735000 31036 ts_tablet_manager.cc:1918] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:57:12.735071 31036 log.cc:1196] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/0a63809b68d44d5cbc7c89b004a7de08
I20260928 22:57:12.735280 31036 ts_tablet_manager.cc:1939] T 0a63809b68d44d5cbc7c89b004a7de08 P 6b9ff8c1b7784202a31785075c39e82f: Deleting consensus metadata
I20260928 22:57:12.735517 31014 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 22:57:12.735579 31014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 22:57:12.735626 31014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:12.735654 31014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:12.741636 31014 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:44891
I20260928 22:57:12.741736 31143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:44891 every 8 connection(s)
I20260928 22:57:12.742004 31014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:57:12.743048 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:12.742652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:46557)
0928 22:57:12.742811 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:12.742911 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
W20260928 22:57:12.743294 31144 heartbeater.cc:644] Failed to heartbeat to 127.14.14.190:33333 (0 consecutive failures): Network error: Failed to ping master at 127.14.14.190:33333: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
I20260928 22:57:12.743579 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:12.743491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:49605)
0928 22:57:12.743538 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:12.743562 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 22:57:12.743830 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:12.743783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:41651)
0928 22:57:12.743803 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:12.743819 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 22:57:12.743953 31144 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:57:12.746119 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 31014
I20260928 22:57:12.746381 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:37421
--local_ip_for_outbound_sockets=127.14.14.131
--tserver_master_addrs=127.14.14.190:33333
--webserver_port=37755
--webserver_interface=127.14.14.131
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:12.827778 31147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:12.828034 31147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:12.828078 31147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:12.829272 31147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:12.829339 31147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:12.829386 31147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:57:12.831058 31147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:37421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=37755
--tserver_master_addrs=127.14.14.190:33333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:12.831352 31147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:12.831599 31147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:12.836730 31155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:12.836750 31152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:12.836756 31153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:12.836891 31147 server_base.cc:1034] running on GCE node
I20260928 22:57:12.837261 31147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:12.837529 31147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:12.838712 31147 hybrid_clock.cc:648] HybridClock initialized: now 1790636232838699 us; error 26 us; skew 500 ppm
Sep 28 22:57:12 dist-test-slave-kzfr krb5kdc[29701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636232, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:12.841130 31147 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:57:12.841665 31147 webserver.cc:460] Webserver started at http://127.14.14.131:37755/ using document root <none> and password file <none>
I20260928 22:57:12.841894 31147 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:12.841964 31147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:12.843267 31147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:57:12.843886 31162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:12.844049 31147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:57:12.844146 31147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "95f2a34421ca4c5aa31aeddc782ca1a6"
format_stamp: "Formatted at 2026-09-28 22:56:31 on dist-test-slave-kzfr"
server_key: "b2bad76a58e6aa34a11f8a670413695c"
server_key_iv: "d57b5a41cd613f90f563de4cf5919681"
server_key_version: "encryptionkey@0"
I20260928 22:57:12.844535 31147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:12.868573 31147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:12.870630 31147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:12.870798 31147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:12.871065 31147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:12.871382 31147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:12.871446 31147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:12.871492 31147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:12.871526 31147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:12.877744 31147 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:37421
I20260928 22:57:12.877795 31275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:37421 every 8 connection(s)
I20260928 22:57:12.878177 31147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:57:12.879215 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:12.878852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:51425)
0928 22:57:12.879011 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:12.879079 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
W20260928 22:57:12.879653 31276 heartbeater.cc:644] Failed to heartbeat to 127.14.14.190:33333 (0 consecutive failures): Network error: Failed to ping master at 127.14.14.190:33333: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
I20260928 22:57:12.879959 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:12.879869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:58217)
0928 22:57:12.879916 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:12.879943 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 22:57:12.880203 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:12.880155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:33849)
0928 22:57:12.880174 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:12.880191 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260928 22:57:12.880324 31276 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:57:12.881250 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 31147
I20260928 22:57:13.700538 31280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:13.700265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:53469)
0928 22:57:13.700398 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:13.700444 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260928 22:57:13.744567 31281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:13.744257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:58755)
0928 22:57:13.744502 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:13.744533 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":77,"threads_started":1}
I20260928 22:57:13.880899 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:13.880697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:40445)
0928 22:57:13.880812 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:13.880847 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260928 22:57:14.300772 31283 raft_consensus.cc:482] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:57:14.300889 31283 raft_consensus.cc:504] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:57:14.301285 31283 leader_election.cc:288] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891), 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421)
Sep 28 22:57:14 dist-test-slave-kzfr krb5kdc[29701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636232, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.130@KRBTEST.COM
Sep 28 22:57:14 dist-test-slave-kzfr krb5kdc[29701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636232, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.131@KRBTEST.COM
I20260928 22:57:14.313305 31285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:14.301502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:44891 (local address 127.14.14.129:48697)
0928 22:57:14.301720 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:14.301731 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:14.301775 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:14.302744 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:14.302748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:14.303253 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:14.303534 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:14.303541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:14.305195 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:14.305195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:14.306521 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:14.306524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:14.307331 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:14.307332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:14.307492 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:14.308006 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:57:14.308015 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:14.309477 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:14.311553 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:14.311560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:14.311563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:14.311876 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:14.312135 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:14.312135 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:14.312136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:14.312273 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:14.312754 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:14.312760 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:14.312894 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 22:57:14.312998 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 22:57:14.313196 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":95,"thread_start_us":59,"threads_started":1}
I20260928 22:57:14.313374 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:14.301656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:45931 (local address 127.14.14.131:37421)
0928 22:57:14.301813 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:14.301825 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:14.301856 (+ 31us) server_negotiation.cc:407] Connection header received
0928 22:57:14.302432 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:14.302435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:14.302510 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:14.302673 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:14.303665 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:14.305060 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:14.306612 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:14.307232 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:14.307346 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:14.309903 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:14.309909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:14.309910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:14.309925 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:14.311822 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:14.312258 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:14.312261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:14.312262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:14.312365 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:14.312575 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:14.312575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:14.312576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:14.312848 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:14.312930 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:14.313145 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 22:57:14.313265 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":52,"threads_started":1}
I20260928 22:57:14.313407 31284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:14.301500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:37421 (local address 127.14.14.129:45931)
0928 22:57:14.301635 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:14.301645 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:14.301746 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:14.302744 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:14.302748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:14.303252 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:14.303534 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:14.303541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:14.305182 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:14.305184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:14.306521 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:14.306524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:14.307468 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:14.307469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:14.307552 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:14.308168 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:57:14.308173 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:14.309802 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:14.311935 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:14.311936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:14.311937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:14.312163 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:14.312433 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:14.312434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:14.312434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:14.312497 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:14.312919 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:14.312920 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:14.312987 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 22:57:14.313057 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 22:57:14.313278 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":152,"thread_start_us":54,"threads_started":1}
I20260928 22:57:14.313328 31286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:14.301648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:48697 (local address 127.14.14.130:44891)
0928 22:57:14.301813 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:14.301825 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:14.301850 (+ 25us) server_negotiation.cc:407] Connection header received
0928 22:57:14.302453 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:14.302456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:14.302510 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:14.302673 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:14.303665 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:14.305059 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:14.306613 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:14.307232 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:14.307347 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:14.309646 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:14.309655 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:14.309655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:14.309665 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:14.311434 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:14.311954 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:14.311957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:14.311958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:14.312059 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:14.312342 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:14.312342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:14.312343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:14.312682 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:14.312743 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:14.313068 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 22:57:14.313217 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
W20260928 22:57:14.314325 30897 leader_election.cc:341] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421): Not found: Tablet not found: 0a63809b68d44d5cbc7c89b004a7de08
W20260928 22:57:14.314421 30896 leader_election.cc:341] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891): Not found: Tablet not found: 0a63809b68d44d5cbc7c89b004a7de08
I20260928 22:57:14.314479 30896 leader_election.cc:302] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d35c93b8f2e24aa7b7cf3a988167eca1; no voters: 6b9ff8c1b7784202a31785075c39e82f, 95f2a34421ca4c5aa31aeddc782ca1a6
I20260928 22:57:14.314673 31283 raft_consensus.cc:2726] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:57:14.701184 31285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:14.701058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:33787)
0928 22:57:14.701111 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:14.701145 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 22:57:14.745201 31288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:14.745013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:43475)
0928 22:57:14.745131 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:14.745167 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260928 22:57:14.881690 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:14.881445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:46589)
0928 22:57:14.881571 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:14.881653 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260928 22:57:15.702055 31290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:15.701835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:48681)
0928 22:57:15.701974 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:15.702008 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260928 22:57:15.745936 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:15.745737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:37837)
0928 22:57:15.745857 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:15.745896 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260928 22:57:15.882334 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:15.882133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:48143)
0928 22:57:15.882258 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:15.882295 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260928 22:57:16.064302 31293 raft_consensus.cc:482] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:57:16.064411 31293 raft_consensus.cc:504] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:57:16.064612 31293 leader_election.cc:288] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891), 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421)
W20260928 22:57:16.065052 30896 leader_election.cc:341] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891): Not found: Tablet not found: 0a63809b68d44d5cbc7c89b004a7de08
W20260928 22:57:16.065135 30897 leader_election.cc:341] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421): Not found: Tablet not found: 0a63809b68d44d5cbc7c89b004a7de08
I20260928 22:57:16.065246 30897 leader_election.cc:302] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d35c93b8f2e24aa7b7cf3a988167eca1; no voters: 6b9ff8c1b7784202a31785075c39e82f, 95f2a34421ca4c5aa31aeddc782ca1a6
I20260928 22:57:16.065342 31293 raft_consensus.cc:2726] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:57:16.702780 31294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:16.702551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:44163)
0928 22:57:16.702690 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:16.702733 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
I20260928 22:57:16.746587 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:16.746369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:49935)
0928 22:57:16.746513 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:16.746550 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":58,"threads_started":1}
I20260928 22:57:16.882997 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:16.882795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:41269)
0928 22:57:16.882914 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:16.882955 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260928 22:57:17.703548 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:17.703318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:60129)
0928 22:57:17.703454 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:17.703501 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260928 22:57:17.747324 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:17.747088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:47537)
0928 22:57:17.747213 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:17.747276 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260928 22:57:17.883718 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:17.883491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:35829)
0928 22:57:17.883629 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:17.883672 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260928 22:57:17.970330 31300 raft_consensus.cc:482] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:57:17.970425 31300 raft_consensus.cc:504] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:57:17.970649 31300 leader_election.cc:288] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891), 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421)
W20260928 22:57:17.971200 30896 leader_election.cc:341] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891): Not found: Tablet not found: 0a63809b68d44d5cbc7c89b004a7de08
W20260928 22:57:17.971272 30897 leader_election.cc:341] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421): Not found: Tablet not found: 0a63809b68d44d5cbc7c89b004a7de08
I20260928 22:57:17.971335 30897 leader_election.cc:302] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d35c93b8f2e24aa7b7cf3a988167eca1; no voters: 6b9ff8c1b7784202a31785075c39e82f, 95f2a34421ca4c5aa31aeddc782ca1a6
I20260928 22:57:17.971426 31300 raft_consensus.cc:2726] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:57:18.704267 31301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:18.704038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:39681)
0928 22:57:18.704191 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:18.704221 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260928 22:57:18.748013 31302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:18.747778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:36971)
0928 22:57:18.747914 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:18.747951 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260928 22:57:18.884431 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:18.884226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:44447)
0928 22:57:18.884361 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:18.884387 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260928 22:57:19.704990 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:19.704750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:44397)
0928 22:57:19.704900 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:19.704940 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260928 22:57:19.748848 31305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:19.748619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:32973)
0928 22:57:19.748763 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:19.748803 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260928 22:57:19.885135 31306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:19.884912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:43921)
0928 22:57:19.885049 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:19.885092 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260928 22:57:20.503216 31307 raft_consensus.cc:482] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:57:20.503348 31307 raft_consensus.cc:504] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:57:20.503619 31307 leader_election.cc:288] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891), 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421)
W20260928 22:57:20.504048 30897 leader_election.cc:341] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421): Not found: Tablet not found: 0a63809b68d44d5cbc7c89b004a7de08
W20260928 22:57:20.504163 30896 leader_election.cc:341] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891): Not found: Tablet not found: 0a63809b68d44d5cbc7c89b004a7de08
I20260928 22:57:20.504225 30896 leader_election.cc:302] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d35c93b8f2e24aa7b7cf3a988167eca1; no voters: 6b9ff8c1b7784202a31785075c39e82f, 95f2a34421ca4c5aa31aeddc782ca1a6
I20260928 22:57:20.504292 31307 raft_consensus.cc:2726] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:57:20.705871 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:20.705549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:38757)
0928 22:57:20.705768 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:20.705806 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":83,"threads_started":1}
I20260928 22:57:20.749526 31309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:20.749329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:36969)
0928 22:57:20.749455 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:20.749487 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260928 22:57:20.885854 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:20.885655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:34269)
0928 22:57:20.885776 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:20.885810 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260928 22:57:21.706569 31311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:21.706341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:41417)
0928 22:57:21.706480 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:21.706522 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":59,"threads_started":1}
I20260928 22:57:21.750239 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:21.750039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:43003)
0928 22:57:21.750161 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:21.750196 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260928 22:57:21.886533 31313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:21.886284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:36611)
0928 22:57:21.886419 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:21.886453 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260928 22:57:22.707333 31314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:22.707119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:57511)
0928 22:57:22.707249 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:22.707287 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260928 22:57:22.750932 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:22.750736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:52041)
0928 22:57:22.750857 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:22.750891 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260928 22:57:22.887179 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:22.886979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:52805)
0928 22:57:22.887106 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:22.887138 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260928 22:57:23.708107 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:23.707868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:43855)
0928 22:57:23.708015 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:23.708059 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260928 22:57:23.751711 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:23.751400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:53247)
0928 22:57:23.751550 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:23.751618 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":71,"threads_started":1}
I20260928 22:57:23.887866 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:23.887667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:43781)
0928 22:57:23.887786 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:23.887824 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260928 22:57:24.615029 31320 raft_consensus.cc:482] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:57:24.615132 31320 raft_consensus.cc:504] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:57:24.615316 31320 leader_election.cc:288] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891), 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421)
W20260928 22:57:24.615748 30896 leader_election.cc:341] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891): Not found: Tablet not found: 0a63809b68d44d5cbc7c89b004a7de08
W20260928 22:57:24.615825 30897 leader_election.cc:341] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421): Not found: Tablet not found: 0a63809b68d44d5cbc7c89b004a7de08
I20260928 22:57:24.615908 30897 leader_election.cc:302] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d35c93b8f2e24aa7b7cf3a988167eca1; no voters: 6b9ff8c1b7784202a31785075c39e82f, 95f2a34421ca4c5aa31aeddc782ca1a6
I20260928 22:57:24.616008 31320 raft_consensus.cc:2726] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:57:24.708866 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:24.708655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:33283)
0928 22:57:24.708791 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:24.708821 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260928 22:57:24.752453 31322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:24.752195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:59331)
0928 22:57:24.752359 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:24.752397 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
I20260928 22:57:24.888509 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:24.888294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:52829)
0928 22:57:24.888417 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:24.888462 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260928 22:57:25.709556 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:25.709349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:54457)
0928 22:57:25.709471 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:25.709511 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260928 22:57:25.753234 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:25.753004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:60915)
0928 22:57:25.753151 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:25.753188 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":61,"threads_started":1}
I20260928 22:57:25.889204 31326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:25.888999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:34255)
0928 22:57:25.889131 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:25.889162 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260928 22:57:26.413970 31327 raft_consensus.cc:482] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:57:26.414072 31327 raft_consensus.cc:504] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9ff8c1b7784202a31785075c39e82f" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44891 } } peers { permanent_uuid: "95f2a34421ca4c5aa31aeddc782ca1a6" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 37421 } } peers { permanent_uuid: "d35c93b8f2e24aa7b7cf3a988167eca1" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 43421 } }
I20260928 22:57:26.414228 31327 leader_election.cc:288] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891), 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421)
W20260928 22:57:26.414573 30896 leader_election.cc:341] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b9ff8c1b7784202a31785075c39e82f (127.14.14.130:44891): Not found: Tablet not found: 0a63809b68d44d5cbc7c89b004a7de08
W20260928 22:57:26.414686 30897 leader_election.cc:341] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95f2a34421ca4c5aa31aeddc782ca1a6 (127.14.14.131:37421): Not found: Tablet not found: 0a63809b68d44d5cbc7c89b004a7de08
I20260928 22:57:26.414732 30897 leader_election.cc:302] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d35c93b8f2e24aa7b7cf3a988167eca1; no voters: 6b9ff8c1b7784202a31785075c39e82f, 95f2a34421ca4c5aa31aeddc782ca1a6
I20260928 22:57:26.414825 31327 raft_consensus.cc:2726] T 0a63809b68d44d5cbc7c89b004a7de08 P d35c93b8f2e24aa7b7cf3a988167eca1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:57:26.710287 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:26.710046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:56597)
0928 22:57:26.710207 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:26.710239 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260928 22:57:26.754082 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:26.753856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:48941)
0928 22:57:26.753989 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:26.754030 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260928 22:57:26.889925 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:26.889706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:59843)
0928 22:57:26.889829 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:26.889862 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260928 22:57:27.711024 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:27.710803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.129:45371)
0928 22:57:27.710943 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:27.710978 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260928 22:57:27.754768 31332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:27.754578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.130:56243)
0928 22:57:27.754692 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:27.754725 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
I20260928 22:57:27.884034 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 30845
I20260928 22:57:27.888684 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 31014
I20260928 22:57:27.890689 31333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:27.890454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33333 (local address 127.14.14.131:47111)
0928 22:57:27.890587 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:27.890631 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.190:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260928 22:57:27.893155 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 31147
2026-09-28T22:57:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58651 ms)
[----------] 30 tests from SecurityITest (231645 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:27 dist-test-slave-kzfr krb5kdc[31336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:27 dist-test-slave-kzfr krb5kdc[31336](info): set up 2 sockets
Sep 28 22:57:27 dist-test-slave-kzfr krb5kdc[31336](info): commencing operation
krb5kdc: starting...
W20260928 22:57:28.933429 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:29 dist-test-slave-kzfr krb5kdc[31336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:57:29.061818 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44255
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:44255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:29.142531 31349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:29.142762 31349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:29.142800 31349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:29.143929 31349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:29.143993 31349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:29.144017 31349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:29.144039 31349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:29.144088 31349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:29.145555 31349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:44255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:29.145841 31349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:29.146065 31349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:29.151293 31355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:29.151321 31354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:29.151338 31349 server_base.cc:1034] running on GCE node
W20260928 22:57:29.151321 31357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:29.151849 31349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:29.152158 31349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:29.153303 31349 hybrid_clock.cc:648] HybridClock initialized: now 1790636249153290 us; error 40 us; skew 500 ppm
Sep 28 22:57:29 dist-test-slave-kzfr krb5kdc[31336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636249, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:29.155771 31349 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:57:29.156213 31349 webserver.cc:460] Webserver started at http://127.14.14.190:33327/ using document root <none> and password file <none>
I20260928 22:57:29.156383 31349 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:29.156420 31349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:29.156505 31349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:29.157485 31349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "42702aa397b84a68a0f9fbf31592b2a4"
format_stamp: "Formatted at 2026-09-28 22:57:29 on dist-test-slave-kzfr"
server_key: "9e389a76fb99f7443bf8ad1083a83ab5"
server_key_iv: "ea10d48d59024b16420d98f367c29c71"
server_key_version: "encryptionkey@0"
I20260928 22:57:29.157851 31349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "42702aa397b84a68a0f9fbf31592b2a4"
format_stamp: "Formatted at 2026-09-28 22:57:29 on dist-test-slave-kzfr"
server_key: "9e389a76fb99f7443bf8ad1083a83ab5"
server_key_iv: "ea10d48d59024b16420d98f367c29c71"
server_key_version: "encryptionkey@0"
I20260928 22:57:29.159062 31349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:29.159674 31364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:29.159829 31349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:29.159926 31349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "42702aa397b84a68a0f9fbf31592b2a4"
format_stamp: "Formatted at 2026-09-28 22:57:29 on dist-test-slave-kzfr"
server_key: "9e389a76fb99f7443bf8ad1083a83ab5"
server_key_iv: "ea10d48d59024b16420d98f367c29c71"
server_key_version: "encryptionkey@0"
I20260928 22:57:29.160027 31349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:29.177537 31349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:29.179688 31349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:29.179849 31349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:29.183710 31349 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:44255
I20260928 22:57:29.183753 31416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:44255 every 8 connection(s)
I20260928 22:57:29.184171 31349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:29.185084 31417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:29.186455 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 31349
I20260928 22:57:29.186590 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:29.186786 14394 external_mini_cluster.cc:1428] Setting key b412b05cd1b3dd6e11d2873aa982109f
I20260928 22:57:29.187899 31417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4: Bootstrap starting.
I20260928 22:57:29.188606 31417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:29.188896 31417 log.cc:824] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:29.189560 31417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4: No bootstrap required, opened a new log
I20260928 22:57:29.191349 31417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42702aa397b84a68a0f9fbf31592b2a4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44255 } }
I20260928 22:57:29.191504 31417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:29.191552 31417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42702aa397b84a68a0f9fbf31592b2a4, State: Initialized, Role: FOLLOWER
I20260928 22:57:29.191700 31417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42702aa397b84a68a0f9fbf31592b2a4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44255 } }
I20260928 22:57:29.191778 31417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:29.191821 31417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:29.191867 31417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:29.192391 31417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42702aa397b84a68a0f9fbf31592b2a4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44255 } }
I20260928 22:57:29.192507 31417 leader_election.cc:302] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42702aa397b84a68a0f9fbf31592b2a4; no voters:
I20260928 22:57:29.192708 31417 leader_election.cc:288] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:29.192731 31422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:29.192842 31422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 42702aa397b84a68a0f9fbf31592b2a4, State: Running, Role: LEADER
I20260928 22:57:29.192975 31417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 22:57:29 dist-test-slave-kzfr krb5kdc[31336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:57:29.192976 31422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42702aa397b84a68a0f9fbf31592b2a4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44255 } }
I20260928 22:57:29.194212 31423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42702aa397b84a68a0f9fbf31592b2a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42702aa397b84a68a0f9fbf31592b2a4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44255 } } }
I20260928 22:57:29.194212 31424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42702aa397b84a68a0f9fbf31592b2a4. Latest consensus state: current_term: 1 leader_uuid: "42702aa397b84a68a0f9fbf31592b2a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42702aa397b84a68a0f9fbf31592b2a4" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44255 } } }
I20260928 22:57:29.194319 31423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:29.194324 31424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4 [sys.catalog]: This master's current role is: LEADER
W20260928 22:57:29.194898 31436 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:57:29.194958 31436 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:57:29.195040 31437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:29.195231 31437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:29.196918 31437 catalog_manager.cc:1347] Generated new cluster ID: d12b1b38344a4d3ba0df107819407e28
I20260928 22:57:29.196973 31437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:29.198144 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.187366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.14.14.190:44255)
0928 22:57:29.187554 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:29.187574 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:29.187606 (+ 32us) server_negotiation.cc:407] Connection header received
0928 22:57:29.188133 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:29.188141 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:29.188213 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:29.188371 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:29.189103 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.190082 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.190855 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.191252 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.191361 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.194055 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:29.194059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:29.194060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:29.194099 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:29.196399 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:29.196790 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:29.196793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:29.196793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:29.196907 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:29.197147 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:29.197148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:29.197148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:29.197414 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:29.197527 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:29.197792 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 22:57:29.197920 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260928 22:57:29.206777 31437 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:29.207501 31437 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:29.218364 31437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4: Generated new TSK 0
I20260928 22:57:29.218555 31437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:57:29.245664 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44255
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:29.328779 31443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:29.329000 31443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:29.329037 31443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:29.330253 31443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:29.330309 31443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:29.330353 31443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:57:29.331795 31443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:29.332046 31443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:29.332257 31443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:29.337224 31451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:29.337291 31443 server_base.cc:1034] running on GCE node
W20260928 22:57:29.337188 31448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:29.337205 31449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:29.337671 31443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:29.337882 31443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:29.339021 31443 hybrid_clock.cc:648] HybridClock initialized: now 1790636249339004 us; error 31 us; skew 500 ppm
Sep 28 22:57:29 dist-test-slave-kzfr krb5kdc[31336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636249, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:29.341449 31443 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:57:29.341921 31443 webserver.cc:460] Webserver started at http://127.14.14.129:41391/ using document root <none> and password file <none>
I20260928 22:57:29.342195 31443 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:29.342264 31443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:29.342402 31443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:29.343410 31443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "dfcb527f1b01465499576efab8b833b3"
format_stamp: "Formatted at 2026-09-28 22:57:29 on dist-test-slave-kzfr"
server_key: "34be07af6f1afba748c2478d66b3adf9"
server_key_iv: "e67f27728f6324068063cefb549162e2"
server_key_version: "encryptionkey@0"
I20260928 22:57:29.343761 31443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "dfcb527f1b01465499576efab8b833b3"
format_stamp: "Formatted at 2026-09-28 22:57:29 on dist-test-slave-kzfr"
server_key: "34be07af6f1afba748c2478d66b3adf9"
server_key_iv: "e67f27728f6324068063cefb549162e2"
server_key_version: "encryptionkey@0"
I20260928 22:57:29.345048 31443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:57:29.345866 31458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:29.346014 31443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:29.346125 31443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "dfcb527f1b01465499576efab8b833b3"
format_stamp: "Formatted at 2026-09-28 22:57:29 on dist-test-slave-kzfr"
server_key: "34be07af6f1afba748c2478d66b3adf9"
server_key_iv: "e67f27728f6324068063cefb549162e2"
server_key_version: "encryptionkey@0"
I20260928 22:57:29.346196 31443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:29.355789 31443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:29.358103 31443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:29.358253 31443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:29.358477 31443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:29.358769 31443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:29.358803 31443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:29.358853 31443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:29.358880 31443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:29.364576 31443 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:41769
I20260928 22:57:29.364650 31571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:41769 every 8 connection(s)
I20260928 22:57:29.364945 31443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:57:29.370748 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 31443
I20260928 22:57:29.370829 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
Sep 28 22:57:29 dist-test-slave-kzfr krb5kdc[31336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636249, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:57:29.371060 14394 external_mini_cluster.cc:1428] Setting key 1e942d854530d18d62e86da74c9987d3
I20260928 22:57:29.374395 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.365622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:43609 (local address 127.14.14.190:44255)
0928 22:57:29.365663 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:29.365665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:29.365899 (+ 234us) server_negotiation.cc:407] Connection header received
0928 22:57:29.366409 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:29.366410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:29.366423 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:29.366470 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:29.367385 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.367897 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.368893 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.369125 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.369207 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.371504 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:29.371506 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:29.371506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:29.371513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:29.372949 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:29.373368 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:29.373369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:29.373370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:29.373434 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:29.373743 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:29.373744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:29.373745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:29.373863 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:29.373934 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:29.374216 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 22:57:29.374306 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:57:29.374632 31574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.365641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44255 (local address 127.14.14.129:43609)
0928 22:57:29.365786 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:29.365819 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:29.366302 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:29.366563 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:29.366566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:29.367107 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:29.367288 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.367292 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.367994 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.367995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.368803 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.368805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.369219 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.369220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.369340 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.369884 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:57:29.369891 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:29.371430 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:29.373057 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:29.373065 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:29.373068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:29.373294 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:29.373498 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:29.373499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:29.373499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:29.373678 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:29.373937 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:29.373944 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:29.374086 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 22:57:29.374182 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 22:57:29.374338 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260928 22:57:29.375084 31572 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44255
I20260928 22:57:29.375173 31572 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:29.375371 31572 heartbeater.cc:511] Master 127.14.14.190:44255 requested a full tablet report, sending...
I20260928 22:57:29.375864 31381 ts_manager.cc:194] Registered new tserver with Master: dfcb527f1b01465499576efab8b833b3 (127.14.14.129:41769)
I20260928 22:57:29.376473 31381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:43609
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:57:29.397945 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44255
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:29.480860 31577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:29.481077 31577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:29.481103 31577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:29.482467 31577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:29.482514 31577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:29.482544 31577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:57:29.484141 31577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:29.484405 31577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:29.484689 31577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:29.490828 31585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:29.490806 31582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:29.490806 31583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:29.491147 31577 server_base.cc:1034] running on GCE node
I20260928 22:57:29.491387 31577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:29.491665 31577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:29.492822 31577 hybrid_clock.cc:648] HybridClock initialized: now 1790636249492806 us; error 30 us; skew 500 ppm
Sep 28 22:57:29 dist-test-slave-kzfr krb5kdc[31336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636249, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:29.495823 31577 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:57:29.496321 31577 webserver.cc:460] Webserver started at http://127.14.14.130:33169/ using document root <none> and password file <none>
I20260928 22:57:29.496532 31577 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:29.496578 31577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:29.496675 31577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:29.497840 31577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "ca4adfed971247ea9feaf08fb27b3f32"
format_stamp: "Formatted at 2026-09-28 22:57:29 on dist-test-slave-kzfr"
server_key: "599986bb0d16a5b3ba8344997ed7b24c"
server_key_iv: "9c210175a7cd815c3ed67271233813e7"
server_key_version: "encryptionkey@0"
I20260928 22:57:29.498171 31577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "ca4adfed971247ea9feaf08fb27b3f32"
format_stamp: "Formatted at 2026-09-28 22:57:29 on dist-test-slave-kzfr"
server_key: "599986bb0d16a5b3ba8344997ed7b24c"
server_key_iv: "9c210175a7cd815c3ed67271233813e7"
server_key_version: "encryptionkey@0"
I20260928 22:57:29.499593 31577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:57:29.500306 31592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:29.500537 31577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:29.500651 31577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "ca4adfed971247ea9feaf08fb27b3f32"
format_stamp: "Formatted at 2026-09-28 22:57:29 on dist-test-slave-kzfr"
server_key: "599986bb0d16a5b3ba8344997ed7b24c"
server_key_iv: "9c210175a7cd815c3ed67271233813e7"
server_key_version: "encryptionkey@0"
I20260928 22:57:29.500730 31577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:29.515900 31577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:29.517944 31577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:29.518100 31577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:29.518329 31577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:29.518651 31577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:29.518702 31577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:29.518743 31577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:29.518769 31577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:29.524612 31577 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:38925
I20260928 22:57:29.524654 31705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:38925 every 8 connection(s)
I20260928 22:57:29.524956 31577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:57:29 dist-test-slave-kzfr krb5kdc[31336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636249, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:57:29.532680 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 31577
I20260928 22:57:29.532790 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:57:29.533080 14394 external_mini_cluster.cc:1428] Setting key 73b3ac91273c8f9990a96eb354fd9866
I20260928 22:57:29.534073 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.525671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:56009 (local address 127.14.14.190:44255)
0928 22:57:29.525717 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:29.525719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:29.525915 (+ 196us) server_negotiation.cc:407] Connection header received
0928 22:57:29.526435 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:29.526436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:29.526449 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:29.526510 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:29.527481 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.527998 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.528962 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.529175 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.529257 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.531399 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:29.531400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:29.531400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:29.531407 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:29.532513 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:29.532981 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:29.532982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:29.532982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:29.533034 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:29.533303 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:29.533304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:29.533305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:29.533452 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:29.533520 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:29.533900 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 22:57:29.533983 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:57:29.534266 31708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.525643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44255 (local address 127.14.14.130:56009)
0928 22:57:29.525797 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:29.525832 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:29.526331 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:29.526591 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:29.526595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:29.527174 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:29.527372 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.527376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.528089 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.528090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.528873 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.528875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.529266 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.529268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.529390 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.529911 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:57:29.529919 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:29.531311 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:29.532589 (+ 1278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:29.532596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:29.532598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:29.532909 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:29.533102 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:29.533103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:29.533103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:29.533238 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:29.533527 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:29.533532 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:29.533756 (+ 224us) client_negotiation.cc:769] Sending connection context
0928 22:57:29.533877 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 22:57:29.534040 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260928 22:57:29.534789 31706 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44255
I20260928 22:57:29.534893 31706 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:29.535077 31706 heartbeater.cc:511] Master 127.14.14.190:44255 requested a full tablet report, sending...
I20260928 22:57:29.535411 31381 ts_manager.cc:194] Registered new tserver with Master: ca4adfed971247ea9feaf08fb27b3f32 (127.14.14.130:38925)
I20260928 22:57:29.535815 31381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:56009
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:57:29.560786 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44255
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:29.643013 31711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:29.643216 31711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:29.643239 31711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:29.644304 31711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:29.644339 31711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:29.644366 31711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:57:29.645831 31711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:29.646027 31711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:29.646201 31711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:29.651249 31719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:29.651266 31717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:29.651340 31711 server_base.cc:1034] running on GCE node
W20260928 22:57:29.651270 31716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:29.651669 31711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:29.651906 31711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:29.653047 31711 hybrid_clock.cc:648] HybridClock initialized: now 1790636249653022 us; error 43 us; skew 500 ppm
Sep 28 22:57:29 dist-test-slave-kzfr krb5kdc[31336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636249, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:29.655637 31711 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:57:29.656085 31711 webserver.cc:460] Webserver started at http://127.14.14.131:34367/ using document root <none> and password file <none>
I20260928 22:57:29.656312 31711 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:29.656355 31711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:29.656499 31711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:29.657526 31711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "2a259f1873ca4cc69aaef6a6365b6d18"
format_stamp: "Formatted at 2026-09-28 22:57:29 on dist-test-slave-kzfr"
server_key: "4f9aa5d13de7f2b4a0a2eb6357324388"
server_key_iv: "c1d1314346c010af3b2c4313f78b1255"
server_key_version: "encryptionkey@0"
I20260928 22:57:29.657933 31711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "2a259f1873ca4cc69aaef6a6365b6d18"
format_stamp: "Formatted at 2026-09-28 22:57:29 on dist-test-slave-kzfr"
server_key: "4f9aa5d13de7f2b4a0a2eb6357324388"
server_key_iv: "c1d1314346c010af3b2c4313f78b1255"
server_key_version: "encryptionkey@0"
I20260928 22:57:29.659226 31711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:29.659953 31726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:29.660143 31711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:29.660238 31711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "2a259f1873ca4cc69aaef6a6365b6d18"
format_stamp: "Formatted at 2026-09-28 22:57:29 on dist-test-slave-kzfr"
server_key: "4f9aa5d13de7f2b4a0a2eb6357324388"
server_key_iv: "c1d1314346c010af3b2c4313f78b1255"
server_key_version: "encryptionkey@0"
I20260928 22:57:29.660315 31711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:29.678272 31711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:29.680567 31711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:29.680735 31711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:29.680970 31711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:29.681293 31711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:29.681344 31711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:29.681391 31711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:29.681419 31711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:29.687568 31711 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:33045
I20260928 22:57:29.687629 31839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:33045 every 8 connection(s)
I20260928 22:57:29.687934 31711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:57:29 dist-test-slave-kzfr krb5kdc[31336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636249, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:57:29.696044 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 31711
I20260928 22:57:29.696172 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:57:29.696372 14394 external_mini_cluster.cc:1428] Setting key 65b08ffb17cdd89e8a88c1497d1869a2
I20260928 22:57:29.697713 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.688653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:44419 (local address 127.14.14.190:44255)
0928 22:57:29.688722 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:29.688724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:29.688927 (+ 203us) server_negotiation.cc:407] Connection header received
0928 22:57:29.689487 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:29.689488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:29.689498 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:29.689569 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:29.690591 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.691084 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.692061 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.692238 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.692287 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.694707 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:29.694709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:29.694709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:29.694718 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:29.696175 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:29.696661 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:29.696663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:29.696663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:29.696730 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:29.697024 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:29.697025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:29.697025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:29.697154 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:29.697240 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:29.697510 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 22:57:29.697633 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:57:29.697873 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.688622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44255 (local address 127.14.14.131:44419)
0928 22:57:29.688794 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:29.688836 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:29.689341 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:29.689704 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:29.689708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:29.690277 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:29.690475 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.690479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.691160 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.691160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.691988 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.691990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.692303 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.692304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.692399 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.692966 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:57:29.692974 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:29.694604 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:29.696256 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:29.696262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:29.696264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:29.696567 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:29.696799 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:29.696800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:29.696801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:29.696960 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:29.697238 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:29.697245 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:29.697401 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 22:57:29.697487 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 22:57:29.697665 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260928 22:57:29.698302 31840 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44255
I20260928 22:57:29.698392 31840 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:29.698539 31840 heartbeater.cc:511] Master 127.14.14.190:44255 requested a full tablet report, sending...
I20260928 22:57:29.698920 31381 ts_manager.cc:194] Registered new tserver with Master: 2a259f1873ca4cc69aaef6a6365b6d18 (127.14.14.131:33045)
I20260928 22:57:29.699291 31381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:44419
I20260928 22:57:29.708150 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:57:29.715322 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.709504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.14.14.190:44255)
0928 22:57:29.709566 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:29.709568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:29.709795 (+ 227us) server_negotiation.cc:407] Connection header received
0928 22:57:29.709835 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:29.709836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:29.709845 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:29.709886 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:29.710407 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.710875 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.711588 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.711765 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.711863 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.713025 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:29.713026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:29.713026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:29.713032 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:29.714434 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:29.714727 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:29.714728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:29.714728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:29.714776 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:29.714917 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:29.714917 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:29.714917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:29.715062 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:29.715130 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:29.715209 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 22:57:29.715263 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:57:29.716871 31381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:57:29.718881 31381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:57:29.725342 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.721298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:38925 (local address 127.0.0.1:52070)
0928 22:57:29.721551 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:29.721561 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:29.721633 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:29.722054 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:29.722055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:29.722056 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:29.722091 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.722092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.722933 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.722935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.724147 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.724150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.725045 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.725046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.725130 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.725147 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 22:57:29.725214 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 22:57:29.725270 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":93,"threads_started":1}
I20260928 22:57:29.725540 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.721174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.14.14.131:33045)
0928 22:57:29.721682 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:29.721687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:29.721697 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:57:29.721733 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:29.721735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:29.721750 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:29.721998 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:29.722069 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.723164 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.724382 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.725071 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.725188 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.725243 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:29.725355 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 22:57:29.725395 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":79,"threads_started":1}
I20260928 22:57:29.725936 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.721615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52070 (local address 127.14.14.130:38925)
0928 22:57:29.721831 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:29.721836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:29.721846 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:57:29.721883 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:29.721885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:29.721902 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:29.721976 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:29.722168 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.722838 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.724236 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.724950 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.725034 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.725096 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:29.725807 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 22:57:29.725851 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":95,"threads_started":1}
I20260928 22:57:29.725456 31852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.721091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:33045 (local address 127.0.0.1:56344)
0928 22:57:29.721222 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:29.721262 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:29.721322 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:29.721847 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:29.721848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:29.721851 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:29.721906 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.721908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.723270 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.723271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.724313 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.724315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.725192 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.725194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.725259 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.725263 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:57:29.725340 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 22:57:29.725385 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260928 22:57:29.726409 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.721265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.14.14.129:41769)
0928 22:57:29.721442 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:29.721448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:29.721500 (+ 52us) server_negotiation.cc:407] Connection header received
0928 22:57:29.721601 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:29.721603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:29.721621 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:29.721964 (+ 343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:29.722038 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.723164 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.725171 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.725914 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.726008 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.726071 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:29.726285 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 22:57:29.726332 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260928 22:57:29.726521 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.721093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:41769 (local address 127.0.0.1:38228)
0928 22:57:29.721443 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:29.721453 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:29.721515 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:29.721764 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:29.721767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:29.721769 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:29.721852 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.721854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.724381 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.724382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.725079 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.725080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.726106 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.726108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.726190 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.726193 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:57:29.726263 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 22:57:29.726306 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":79,"threads_started":1}
I20260928 22:57:29.726778 31640 tablet_service.cc:1467] Processing CreateTablet for tablet 712f96934ebf47cfb171c526e66afbef (DEFAULT_TABLE table=test-table [id=2fc8beb30c034377bbbdcc133f4c1f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:29.727160 31640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 712f96934ebf47cfb171c526e66afbef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:29.727108 31774 tablet_service.cc:1467] Processing CreateTablet for tablet 712f96934ebf47cfb171c526e66afbef (DEFAULT_TABLE table=test-table [id=2fc8beb30c034377bbbdcc133f4c1f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:29.727394 31774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 712f96934ebf47cfb171c526e66afbef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:29.729929 31506 tablet_service.cc:1467] Processing CreateTablet for tablet 712f96934ebf47cfb171c526e66afbef (DEFAULT_TABLE table=test-table [id=2fc8beb30c034377bbbdcc133f4c1f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:29.730250 31506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 712f96934ebf47cfb171c526e66afbef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:29.730587 31858 tablet_bootstrap.cc:492] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32: Bootstrap starting.
I20260928 22:57:29.731242 31858 tablet_bootstrap.cc:654] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:29.731846 31858 log.cc:824] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:29.732187 31857 tablet_bootstrap.cc:492] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18: Bootstrap starting.
I20260928 22:57:29.732545 31858 tablet_bootstrap.cc:492] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32: No bootstrap required, opened a new log
I20260928 22:57:29.732632 31858 ts_tablet_manager.cc:1397] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:57:29.732904 31857 tablet_bootstrap.cc:654] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:29.733439 31857 log.cc:824] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:29.734017 31860 tablet_bootstrap.cc:492] T 712f96934ebf47cfb171c526e66afbef P dfcb527f1b01465499576efab8b833b3: Bootstrap starting.
I20260928 22:57:29.734380 31857 tablet_bootstrap.cc:492] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18: No bootstrap required, opened a new log
I20260928 22:57:29.734467 31857 ts_tablet_manager.cc:1397] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:57:29.734472 31858 raft_consensus.cc:348] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb527f1b01465499576efab8b833b3" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41769 } } peers { permanent_uuid: "2a259f1873ca4cc69aaef6a6365b6d18" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33045 } } peers { permanent_uuid: "ca4adfed971247ea9feaf08fb27b3f32" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 38925 } }
I20260928 22:57:29.734656 31858 raft_consensus.cc:374] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:29.734712 31860 tablet_bootstrap.cc:654] T 712f96934ebf47cfb171c526e66afbef P dfcb527f1b01465499576efab8b833b3: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:29.734719 31858 raft_consensus.cc:729] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4adfed971247ea9feaf08fb27b3f32, State: Initialized, Role: FOLLOWER
I20260928 22:57:29.734874 31858 consensus_queue.cc:260] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb527f1b01465499576efab8b833b3" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41769 } } peers { permanent_uuid: "2a259f1873ca4cc69aaef6a6365b6d18" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33045 } } peers { permanent_uuid: "ca4adfed971247ea9feaf08fb27b3f32" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 38925 } }
I20260928 22:57:29.735141 31858 ts_tablet_manager.cc:1428] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:57:29.735229 31860 log.cc:824] T 712f96934ebf47cfb171c526e66afbef P dfcb527f1b01465499576efab8b833b3: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:29.735280 31706 heartbeater.cc:503] Master 127.14.14.190:44255 was elected leader, sending a full tablet report...
I20260928 22:57:29.736033 31860 tablet_bootstrap.cc:492] T 712f96934ebf47cfb171c526e66afbef P dfcb527f1b01465499576efab8b833b3: No bootstrap required, opened a new log
I20260928 22:57:29.736104 31860 ts_tablet_manager.cc:1397] T 712f96934ebf47cfb171c526e66afbef P dfcb527f1b01465499576efab8b833b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:57:29.736562 31857 raft_consensus.cc:348] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb527f1b01465499576efab8b833b3" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41769 } } peers { permanent_uuid: "2a259f1873ca4cc69aaef6a6365b6d18" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33045 } } peers { permanent_uuid: "ca4adfed971247ea9feaf08fb27b3f32" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 38925 } }
I20260928 22:57:29.736675 31857 raft_consensus.cc:374] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:29.736694 31857 raft_consensus.cc:729] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a259f1873ca4cc69aaef6a6365b6d18, State: Initialized, Role: FOLLOWER
I20260928 22:57:29.736797 31857 consensus_queue.cc:260] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb527f1b01465499576efab8b833b3" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41769 } } peers { permanent_uuid: "2a259f1873ca4cc69aaef6a6365b6d18" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33045 } } peers { permanent_uuid: "ca4adfed971247ea9feaf08fb27b3f32" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 38925 } }
I20260928 22:57:29.737063 31857 ts_tablet_manager.cc:1428] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:57:29.737111 31840 heartbeater.cc:503] Master 127.14.14.190:44255 was elected leader, sending a full tablet report...
I20260928 22:57:29.737835 31860 raft_consensus.cc:348] T 712f96934ebf47cfb171c526e66afbef P dfcb527f1b01465499576efab8b833b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb527f1b01465499576efab8b833b3" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41769 } } peers { permanent_uuid: "2a259f1873ca4cc69aaef6a6365b6d18" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33045 } } peers { permanent_uuid: "ca4adfed971247ea9feaf08fb27b3f32" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 38925 } }
I20260928 22:57:29.738065 31860 raft_consensus.cc:374] T 712f96934ebf47cfb171c526e66afbef P dfcb527f1b01465499576efab8b833b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:29.738109 31860 raft_consensus.cc:729] T 712f96934ebf47cfb171c526e66afbef P dfcb527f1b01465499576efab8b833b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcb527f1b01465499576efab8b833b3, State: Initialized, Role: FOLLOWER
I20260928 22:57:29.738207 31860 consensus_queue.cc:260] T 712f96934ebf47cfb171c526e66afbef P dfcb527f1b01465499576efab8b833b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb527f1b01465499576efab8b833b3" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41769 } } peers { permanent_uuid: "2a259f1873ca4cc69aaef6a6365b6d18" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33045 } } peers { permanent_uuid: "ca4adfed971247ea9feaf08fb27b3f32" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 38925 } }
I20260928 22:57:29.738461 31860 ts_tablet_manager.cc:1428] T 712f96934ebf47cfb171c526e66afbef P dfcb527f1b01465499576efab8b833b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:57:29.738566 31572 heartbeater.cc:503] Master 127.14.14.190:44255 was elected leader, sending a full tablet report...
W20260928 22:57:29.775498 31707 tablet.cc:2369] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:57:29.825435 31862 raft_consensus.cc:482] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:57:29.825560 31862 raft_consensus.cc:504] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb527f1b01465499576efab8b833b3" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41769 } } peers { permanent_uuid: "2a259f1873ca4cc69aaef6a6365b6d18" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33045 } } peers { permanent_uuid: "ca4adfed971247ea9feaf08fb27b3f32" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 38925 } }
I20260928 22:57:29.825959 31862 leader_election.cc:288] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfcb527f1b01465499576efab8b833b3 (127.14.14.129:41769), 2a259f1873ca4cc69aaef6a6365b6d18 (127.14.14.131:33045)
I20260928 22:57:29.828835 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.826283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:44057 (local address 127.14.14.129:41769)
0928 22:57:29.826333 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:29.826335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:29.826437 (+ 102us) server_negotiation.cc:407] Connection header received
0928 22:57:29.826495 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:29.826496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:29.826508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:29.826566 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:29.826741 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.827212 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.828123 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.828501 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.828588 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.828595 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:29.828722 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 22:57:29.828764 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:57:29.828840 31866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.826148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:41769 (local address 127.14.14.130:44057)
0928 22:57:29.826369 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:29.826377 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:29.826430 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:29.826576 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:29.826577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:29.826579 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:29.826671 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.826672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.827290 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.827291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.828027 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.828029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.828581 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.828582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.828651 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.828660 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 22:57:29.828705 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 22:57:29.828736 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":19,"thread_start_us":120,"threads_started":1}
I20260928 22:57:29.828913 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.826245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:35153 (local address 127.14.14.131:33045)
0928 22:57:29.826305 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:29.826307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:29.826314 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:57:29.826345 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:29.826346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:29.826355 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:29.826417 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:29.826625 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.827153 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.828118 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.828497 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.828618 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.828627 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:29.828802 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 22:57:29.828841 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:57:29.828953 31708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.826134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:33045 (local address 127.14.14.130:35153)
0928 22:57:29.826189 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:29.826202 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:29.826254 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:29.826454 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:29.826455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:29.826457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:29.826534 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.826537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.827231 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.827232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.828024 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:29.828027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.828631 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.828632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:29.828728 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.828731 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:57:29.828786 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:57:29.828823 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":1}
I20260928 22:57:29.829210 31526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712f96934ebf47cfb171c526e66afbef" candidate_uuid: "ca4adfed971247ea9feaf08fb27b3f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcb527f1b01465499576efab8b833b3" is_pre_election: true
I20260928 22:57:29.829263 31794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712f96934ebf47cfb171c526e66afbef" candidate_uuid: "ca4adfed971247ea9feaf08fb27b3f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a259f1873ca4cc69aaef6a6365b6d18" is_pre_election: true
I20260928 22:57:29.829380 31526 raft_consensus.cc:2454] T 712f96934ebf47cfb171c526e66afbef P dfcb527f1b01465499576efab8b833b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca4adfed971247ea9feaf08fb27b3f32 in term 0.
I20260928 22:57:29.829416 31794 raft_consensus.cc:2454] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca4adfed971247ea9feaf08fb27b3f32 in term 0.
I20260928 22:57:29.829655 31594 leader_election.cc:302] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca4adfed971247ea9feaf08fb27b3f32, dfcb527f1b01465499576efab8b833b3; no voters:
I20260928 22:57:29.829845 31862 raft_consensus.cc:2781] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:57:29.829897 31862 raft_consensus.cc:482] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:57:29.829931 31862 raft_consensus.cc:3037] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:29.830672 31862 raft_consensus.cc:504] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb527f1b01465499576efab8b833b3" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41769 } } peers { permanent_uuid: "2a259f1873ca4cc69aaef6a6365b6d18" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33045 } } peers { permanent_uuid: "ca4adfed971247ea9feaf08fb27b3f32" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 38925 } }
I20260928 22:57:29.830821 31862 leader_election.cc:288] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [CANDIDATE]: Term 1 election: Requested vote from peers dfcb527f1b01465499576efab8b833b3 (127.14.14.129:41769), 2a259f1873ca4cc69aaef6a6365b6d18 (127.14.14.131:33045)
I20260928 22:57:29.830968 31794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712f96934ebf47cfb171c526e66afbef" candidate_uuid: "ca4adfed971247ea9feaf08fb27b3f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a259f1873ca4cc69aaef6a6365b6d18"
I20260928 22:57:29.831018 31526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712f96934ebf47cfb171c526e66afbef" candidate_uuid: "ca4adfed971247ea9feaf08fb27b3f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcb527f1b01465499576efab8b833b3"
I20260928 22:57:29.831055 31794 raft_consensus.cc:3037] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:29.831110 31526 raft_consensus.cc:3037] T 712f96934ebf47cfb171c526e66afbef P dfcb527f1b01465499576efab8b833b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:29.831763 31794 raft_consensus.cc:2454] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca4adfed971247ea9feaf08fb27b3f32 in term 1.
I20260928 22:57:29.831841 31526 raft_consensus.cc:2454] T 712f96934ebf47cfb171c526e66afbef P dfcb527f1b01465499576efab8b833b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca4adfed971247ea9feaf08fb27b3f32 in term 1.
I20260928 22:57:29.831930 31593 leader_election.cc:302] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a259f1873ca4cc69aaef6a6365b6d18, ca4adfed971247ea9feaf08fb27b3f32; no voters:
I20260928 22:57:29.832015 31862 raft_consensus.cc:2781] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:29.832170 31862 raft_consensus.cc:686] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [term 1 LEADER]: Becoming Leader. State: Replica: ca4adfed971247ea9feaf08fb27b3f32, State: Running, Role: LEADER
I20260928 22:57:29.832283 31862 consensus_queue.cc:237] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb527f1b01465499576efab8b833b3" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41769 } } peers { permanent_uuid: "2a259f1873ca4cc69aaef6a6365b6d18" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33045 } } peers { permanent_uuid: "ca4adfed971247ea9feaf08fb27b3f32" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 38925 } }
I20260928 22:57:29.833129 31381 catalog_manager.cc:5495] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca4adfed971247ea9feaf08fb27b3f32 (127.14.14.130). New cstate: current_term: 1 leader_uuid: "ca4adfed971247ea9feaf08fb27b3f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb527f1b01465499576efab8b833b3" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a259f1873ca4cc69aaef6a6365b6d18" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca4adfed971247ea9feaf08fb27b3f32" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 38925 } health_report { overall_health: HEALTHY } } }
I20260928 22:57:29.841007 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.838885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52078 (local address 127.14.14.130:38925)
0928 22:57:29.838940 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:29.838942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:29.838949 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:57:29.838990 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:29.838991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:29.839006 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:29.839068 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:57:29.839160 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.839592 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.840361 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:29.840562 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:29.840620 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:29.840762 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:57:29.840850 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:57:29.840906 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:29.840939 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 22:57:29.840953 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:57:29.842859 31526 raft_consensus.cc:1261] T 712f96934ebf47cfb171c526e66afbef P dfcb527f1b01465499576efab8b833b3 [term 1 FOLLOWER]: Refusing update from remote peer ca4adfed971247ea9feaf08fb27b3f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:57:29.842859 31794 raft_consensus.cc:1261] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18 [term 1 FOLLOWER]: Refusing update from remote peer ca4adfed971247ea9feaf08fb27b3f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:57:29.843226 31867 consensus_queue.cc:1035] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcb527f1b01465499576efab8b833b3" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:57:29.843348 31862 consensus_queue.cc:1035] T 712f96934ebf47cfb171c526e66afbef P ca4adfed971247ea9feaf08fb27b3f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a259f1873ca4cc69aaef6a6365b6d18" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:57:29.845916 31872 mvcc.cc:205] Tried to move back new op lower bound from 7334446079353425920 to 7334446079313641472. Current Snapshot: MvccSnapshot[applied={T|T < 7334446079353425920}]
I20260928 22:57:29.848018 31381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:57:29.848096 31381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:57:29.848821 31381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42702aa397b84a68a0f9fbf31592b2a4: Sending DeleteTablet for 3 replicas of tablet 712f96934ebf47cfb171c526e66afbef
I20260928 22:57:29.849148 31774 tablet_service.cc:1514] Processing DeleteTablet for tablet 712f96934ebf47cfb171c526e66afbef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:29 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:56344
I20260928 22:57:29.849182 31640 tablet_service.cc:1514] Processing DeleteTablet for tablet 712f96934ebf47cfb171c526e66afbef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:29 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:52070
I20260928 22:57:29.849355 31506 tablet_service.cc:1514] Processing DeleteTablet for tablet 712f96934ebf47cfb171c526e66afbef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:29 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:38228
I20260928 22:57:29.849524 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 31443
I20260928 22:57:29.850075 31879 tablet_replica.cc:330] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18: stopping tablet replica
I20260928 22:57:29.850160 31879 raft_consensus.cc:2229] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:57:29.850229 31879 raft_consensus.cc:2258] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:57:29.850590 31879 ts_tablet_manager.cc:1905] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:57:29.852142 31879 ts_tablet_manager.cc:1918] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:57:29.852196 31879 log.cc:1196] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/712f96934ebf47cfb171c526e66afbef
I20260928 22:57:29.852389 31879 ts_tablet_manager.cc:1939] T 712f96934ebf47cfb171c526e66afbef P 2a259f1873ca4cc69aaef6a6365b6d18: Deleting consensus metadata
I20260928 22:57:29.852757 31365 catalog_manager.cc:4841] TS 2a259f1873ca4cc69aaef6a6365b6d18 (127.14.14.131:33045): tablet 712f96934ebf47cfb171c526e66afbef (table test-table [id=2fc8beb30c034377bbbdcc133f4c1f52]) successfully deleted
W20260928 22:57:29.856113 31366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:41769 (error 108)
I20260928 22:57:29.856302 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 31577
I20260928 22:57:29.857074 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.856941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:41769 (local address 127.0.0.1:38234)
0928 22:57:29.856998 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:29.857035 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:41769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 22:57:29.857409 31366 catalog_manager.cc:4584] TS dfcb527f1b01465499576efab8b833b3 (127.14.14.129:41769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 712f96934ebf47cfb171c526e66afbef: Network error: Client connection negotiation failed: client connection to 127.14.14.129:41769: connect: Connection refused (error 111)
I20260928 22:57:29.861351 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 31711
I20260928 22:57:29.861416 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:29.861314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:38925 (local address 127.0.0.1:52086)
0928 22:57:29.861355 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:29.861392 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.130:38925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 22:57:29.865967 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 31349
2026-09-28T22:57:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1972 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T22:57:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:29Z Disabled control of system clock
I20260928 22:57:29.893258 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:36539
--webserver_interface=127.14.14.190
--webserver_port=0
--builtin_ntp_servers=127.14.14.148:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:36539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:57:29.973272 31886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:29.973493 31886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:29.973513 31886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:29.974694 31886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:29.974740 31886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:29.974753 31886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:29.974766 31886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:29.974779 31886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:29.976135 31886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:36539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:36539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:29.976394 31886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:29.976701 31886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:29.981609 31892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:29.981600 31894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:29.981756 31886 server_base.cc:1034] running on GCE node
W20260928 22:57:29.981715 31891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:29.982131 31886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:29.982420 31886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:29.983582 31886 hybrid_clock.cc:648] HybridClock initialized: now 1790636249983568 us; error 32 us; skew 500 ppm
I20260928 22:57:29.984786 31886 webserver.cc:460] Webserver started at http://127.14.14.190:46599/ using document root <none> and password file <none>
I20260928 22:57:29.985025 31886 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:29.985116 31886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:29.985270 31886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:29.986385 31886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "bca998ed4bfc47e29e7d82513e301a31"
format_stamp: "Formatted at 2026-09-28 22:57:29 on dist-test-slave-kzfr"
server_key: "cd4f118158f5879c04585d8a90d17222"
server_key_iv: "fb4e39fd52ed55156e7e498cb9f7f849"
server_key_version: "encryptionkey@0"
I20260928 22:57:29.986796 31886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "bca998ed4bfc47e29e7d82513e301a31"
format_stamp: "Formatted at 2026-09-28 22:57:29 on dist-test-slave-kzfr"
server_key: "cd4f118158f5879c04585d8a90d17222"
server_key_iv: "fb4e39fd52ed55156e7e498cb9f7f849"
server_key_version: "encryptionkey@0"
I20260928 22:57:29.988111 31886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:57:29.988941 31900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:29.989169 31886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:29.989279 31886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "bca998ed4bfc47e29e7d82513e301a31"
format_stamp: "Formatted at 2026-09-28 22:57:29 on dist-test-slave-kzfr"
server_key: "cd4f118158f5879c04585d8a90d17222"
server_key_iv: "fb4e39fd52ed55156e7e498cb9f7f849"
server_key_version: "encryptionkey@0"
I20260928 22:57:29.989358 31886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:29.998698 31886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:29.998988 31886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:29.999138 31886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:30.003079 31886 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:36539
I20260928 22:57:30.003106 31952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:36539 every 8 connection(s)
I20260928 22:57:30.003521 31886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:30.004490 31953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:30.006784 31953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31: Bootstrap starting.
I20260928 22:57:30.007380 31953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:30.007570 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 31886
I20260928 22:57:30.007681 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:30.007727 31953 log.cc:824] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:30.007901 14394 external_mini_cluster.cc:1428] Setting key e7653bab72dfadb62e7277a0bafb5808
I20260928 22:57:30.008606 31953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31: No bootstrap required, opened a new log
I20260928 22:57:30.011596 31953 raft_consensus.cc:348] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca998ed4bfc47e29e7d82513e301a31" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36539 } }
I20260928 22:57:30.011729 31953 raft_consensus.cc:374] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:30.011750 31953 raft_consensus.cc:729] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca998ed4bfc47e29e7d82513e301a31, State: Initialized, Role: FOLLOWER
I20260928 22:57:30.011893 31953 consensus_queue.cc:260] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca998ed4bfc47e29e7d82513e301a31" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36539 } }
I20260928 22:57:30.011973 31953 raft_consensus.cc:388] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:30.012012 31953 raft_consensus.cc:482] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:30.012060 31953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:30.012722 31953 raft_consensus.cc:504] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca998ed4bfc47e29e7d82513e301a31" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36539 } }
I20260928 22:57:30.012826 31953 leader_election.cc:302] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bca998ed4bfc47e29e7d82513e301a31; no voters:
I20260928 22:57:30.012995 31953 leader_election.cc:288] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:30.013087 31958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:30.013267 31953 sys_catalog.cc:564] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:30.013283 31958 raft_consensus.cc:686] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [term 1 LEADER]: Becoming Leader. State: Replica: bca998ed4bfc47e29e7d82513e301a31, State: Running, Role: LEADER
I20260928 22:57:30.013469 31958 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca998ed4bfc47e29e7d82513e301a31" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36539 } }
I20260928 22:57:30.013854 31959 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bca998ed4bfc47e29e7d82513e301a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca998ed4bfc47e29e7d82513e301a31" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36539 } } }
I20260928 22:57:30.013935 31959 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:30.014111 31960 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca998ed4bfc47e29e7d82513e301a31. Latest consensus state: current_term: 1 leader_uuid: "bca998ed4bfc47e29e7d82513e301a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca998ed4bfc47e29e7d82513e301a31" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36539 } } }
I20260928 22:57:30.014165 31960 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:30.014585 31966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:30.015116 31966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:30.015326 31957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.008410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.14.14.190:36539)
0928 22:57:30.008619 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:30.008628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:30.008665 (+ 37us) server_negotiation.cc:407] Connection header received
0928 22:57:30.009297 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:30.009306 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:30.009384 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:30.009656 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:30.009876 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.011349 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.012367 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.013025 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.014033 (+ 1008us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.014191 (+ 158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:30.014194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:30.014194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:30.014400 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:30.014466 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:30.014484 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:30.014839 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:30.014993 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 22:57:30.015072 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260928 22:57:30.016573 31966 catalog_manager.cc:1347] Generated new cluster ID: 5170f1fca119490bbf931be13c8fd283
I20260928 22:57:30.016628 31966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:30.031572 31966 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:30.032408 31966 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:30.038198 31966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31: Generated new TSK 0
I20260928 22:57:30.038440 31966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:30.047616 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36539
--builtin_ntp_servers=127.14.14.148:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:57:30.144657 31977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:30.144889 31977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:30.144927 31977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:30.146157 31977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:30.146211 31977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:30.146255 31977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:57:30.147629 31977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:30.147874 31977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:30.148164 31977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:30.154387 31985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:30.154409 31983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:30.154353 31982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:30.154520 31977 server_base.cc:1034] running on GCE node
I20260928 22:57:30.154829 31977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:30.155119 31977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:30.156316 31977 hybrid_clock.cc:648] HybridClock initialized: now 1790636250156294 us; error 35 us; skew 500 ppm
I20260928 22:57:30.157974 31977 webserver.cc:460] Webserver started at http://127.14.14.129:41379/ using document root <none> and password file <none>
I20260928 22:57:30.158273 31977 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:30.158344 31977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:30.158500 31977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:30.160019 31977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "c47ca03c3a09479ea84d18f7a283ca8e"
format_stamp: "Formatted at 2026-09-28 22:57:30 on dist-test-slave-kzfr"
server_key: "d390332fff3479477a2a877002678a2d"
server_key_iv: "c0108829d8baf4eeb679280975473ed2"
server_key_version: "encryptionkey@0"
I20260928 22:57:30.160495 31977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "c47ca03c3a09479ea84d18f7a283ca8e"
format_stamp: "Formatted at 2026-09-28 22:57:30 on dist-test-slave-kzfr"
server_key: "d390332fff3479477a2a877002678a2d"
server_key_iv: "c0108829d8baf4eeb679280975473ed2"
server_key_version: "encryptionkey@0"
I20260928 22:57:30.162353 31977 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 22:57:30.163252 31991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:30.163434 31977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:30.163563 31977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "c47ca03c3a09479ea84d18f7a283ca8e"
format_stamp: "Formatted at 2026-09-28 22:57:30 on dist-test-slave-kzfr"
server_key: "d390332fff3479477a2a877002678a2d"
server_key_iv: "c0108829d8baf4eeb679280975473ed2"
server_key_version: "encryptionkey@0"
I20260928 22:57:30.163636 31977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:30.171103 31977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:30.171386 31977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:30.171540 31977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:30.171761 31977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:30.172072 31977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:30.172125 31977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:30.172166 31977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:30.172194 31977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:30.178056 31977 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:46137
I20260928 22:57:30.178104 32104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:46137 every 8 connection(s)
I20260928 22:57:30.178496 31977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:57:30.183205 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 31977
I20260928 22:57:30.183343 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:57:30.183591 14394 external_mini_cluster.cc:1428] Setting key f9ba1905d51e536d5000ad5a284da007
I20260928 22:57:30.184774 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36539
--builtin_ntp_servers=127.14.14.148:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:57:30.185220 31957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.179324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:46133 (local address 127.14.14.190:36539)
0928 22:57:30.179405 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:30.179408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:30.179674 (+ 266us) server_negotiation.cc:407] Connection header received
0928 22:57:30.180342 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:30.180343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:30.180352 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:30.180398 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:30.181312 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.181900 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.183467 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.183811 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.183901 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.184887 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:30.184889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:30.184890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:30.184894 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:30.184916 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:30.184920 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:30.184988 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:30.185076 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 22:57:30.185101 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:57:30.185370 32107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.179303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:36539 (local address 127.14.14.129:46133)
0928 22:57:30.179520 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:30.179569 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:30.180232 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:30.180484 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:30.180488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:30.180499 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:30.181212 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.181220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.181990 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.181992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.183363 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.183366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.183910 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.183911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.184099 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.184728 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:57:30.184728 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:30.184768 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:57:30.184792 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:30.184989 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:30.184995 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:30.184999 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:57:30.185072 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:57:30.185164 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260928 22:57:30.186610 32105 heartbeater.cc:347] Connected to a master server at 127.14.14.190:36539
I20260928 22:57:30.186724 32105 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:30.186920 32105 heartbeater.cc:511] Master 127.14.14.190:36539 requested a full tablet report, sending...
I20260928 22:57:30.187480 31917 ts_manager.cc:194] Registered new tserver with Master: c47ca03c3a09479ea84d18f7a283ca8e (127.14.14.129:46137)
I20260928 22:57:30.188476 31917 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.14.129:46133
W20260928 22:57:30.266245 32108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:30.266444 32108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:30.266465 32108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:30.267671 32108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:30.267709 32108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:30.267735 32108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:57:30.269050 32108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:30.269275 32108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:30.269564 32108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:30.274564 32114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:30.274608 32113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:30.274752 32108 server_base.cc:1034] running on GCE node
W20260928 22:57:30.274878 32116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:30.275170 32108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:30.275391 32108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:30.276525 32108 hybrid_clock.cc:648] HybridClock initialized: now 1790636250276514 us; error 31 us; skew 500 ppm
I20260928 22:57:30.277949 32108 webserver.cc:460] Webserver started at http://127.14.14.130:34131/ using document root <none> and password file <none>
I20260928 22:57:30.278173 32108 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:30.278215 32108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:30.278306 32108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:30.279353 32108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512"
format_stamp: "Formatted at 2026-09-28 22:57:30 on dist-test-slave-kzfr"
server_key: "3dce162cdd6a4dac6a975404d2512c43"
server_key_iv: "a49ff4ce53a432bb50d5df46ecf2a2bc"
server_key_version: "encryptionkey@0"
I20260928 22:57:30.279662 32108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512"
format_stamp: "Formatted at 2026-09-28 22:57:30 on dist-test-slave-kzfr"
server_key: "3dce162cdd6a4dac6a975404d2512c43"
server_key_iv: "a49ff4ce53a432bb50d5df46ecf2a2bc"
server_key_version: "encryptionkey@0"
I20260928 22:57:30.280901 32108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:57:30.281634 32122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:30.281813 32108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:30.281884 32108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512"
format_stamp: "Formatted at 2026-09-28 22:57:30 on dist-test-slave-kzfr"
server_key: "3dce162cdd6a4dac6a975404d2512c43"
server_key_iv: "a49ff4ce53a432bb50d5df46ecf2a2bc"
server_key_version: "encryptionkey@0"
I20260928 22:57:30.281939 32108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:30.312417 32108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:30.312716 32108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:30.312820 32108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:30.313010 32108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:30.313422 32108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:30.313460 32108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:30.313627 32108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:30.313661 32108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:30.319519 32108 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:34863
I20260928 22:57:30.319567 32235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:34863 every 8 connection(s)
I20260928 22:57:30.319893 32108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:57:30.320771 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 32108
I20260928 22:57:30.320888 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:57:30.321076 14394 external_mini_cluster.cc:1428] Setting key 17e43c06f740678640bd7e2ef87b0669
I20260928 22:57:30.322032 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36539
--builtin_ntp_servers=127.14.14.148:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:57:30.326130 31957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.320691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:53177 (local address 127.14.14.190:36539)
0928 22:57:30.320744 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:30.320746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:30.320968 (+ 222us) server_negotiation.cc:407] Connection header received
0928 22:57:30.321683 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:30.321684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:30.321695 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:30.321741 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:30.322521 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.323285 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.324697 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.324880 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.324936 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.325878 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:30.325880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:30.325880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:30.325884 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:30.325898 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:30.325901 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:30.325945 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:30.326036 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 22:57:30.326056 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:57:30.326435 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.320553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:36539 (local address 127.14.14.130:53177)
0928 22:57:30.320811 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:30.320852 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:30.321549 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:30.321826 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:30.321830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:30.321840 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:30.322432 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.322437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.323416 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.323418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.324623 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.324625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.324955 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.324956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.325125 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.325721 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:57:30.325722 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:30.325761 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:57:30.325790 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:30.325961 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:30.325967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:30.325970 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:57:30.326038 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 22:57:30.326157 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
I20260928 22:57:30.326918 32236 heartbeater.cc:347] Connected to a master server at 127.14.14.190:36539
I20260928 22:57:30.327044 32236 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:30.327235 32236 heartbeater.cc:511] Master 127.14.14.190:36539 requested a full tablet report, sending...
I20260928 22:57:30.327584 31917 ts_manager.cc:194] Registered new tserver with Master: b52cc2c3cf9e4018afaf6a6bd4da3512 (127.14.14.130:34863)
I20260928 22:57:30.327908 31917 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.14.130:53177
W20260928 22:57:30.405328 32239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:30.405587 32239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:30.405644 32239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:30.406802 32239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:30.406869 32239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:30.406912 32239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:57:30.408260 32239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:30.408527 32239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:30.408823 32239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:30.413849 32244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:30.413856 32247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:30.413950 32245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:30.414075 32239 server_base.cc:1034] running on GCE node
I20260928 22:57:30.414324 32239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:30.414611 32239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:30.415776 32239 hybrid_clock.cc:648] HybridClock initialized: now 1790636250415769 us; error 34 us; skew 500 ppm
I20260928 22:57:30.417073 32239 webserver.cc:460] Webserver started at http://127.14.14.131:33985/ using document root <none> and password file <none>
I20260928 22:57:30.417325 32239 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:30.417403 32239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:30.417528 32239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:30.418610 32239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "080e0d717c56460b9bd6da23205db1f5"
format_stamp: "Formatted at 2026-09-28 22:57:30 on dist-test-slave-kzfr"
server_key: "ee8a9e0078bc927bb563dcaee2ca05a7"
server_key_iv: "2103cef947124c26768289acffb1ee90"
server_key_version: "encryptionkey@0"
I20260928 22:57:30.418897 32239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "080e0d717c56460b9bd6da23205db1f5"
format_stamp: "Formatted at 2026-09-28 22:57:30 on dist-test-slave-kzfr"
server_key: "ee8a9e0078bc927bb563dcaee2ca05a7"
server_key_iv: "2103cef947124c26768289acffb1ee90"
server_key_version: "encryptionkey@0"
I20260928 22:57:30.420122 32239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:57:30.420838 32253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:30.420992 32239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:30.421072 32239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "080e0d717c56460b9bd6da23205db1f5"
format_stamp: "Formatted at 2026-09-28 22:57:30 on dist-test-slave-kzfr"
server_key: "ee8a9e0078bc927bb563dcaee2ca05a7"
server_key_iv: "2103cef947124c26768289acffb1ee90"
server_key_version: "encryptionkey@0"
I20260928 22:57:30.421144 32239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:30.431746 32239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:30.432060 32239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:30.432214 32239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:30.432521 32239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:30.432869 32239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:30.432907 32239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:30.432933 32239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:30.432984 32239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:30.439559 32239 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:35101
I20260928 22:57:30.439637 32366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:35101 every 8 connection(s)
I20260928 22:57:30.439950 32239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:57:30.445339 31957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.440827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:53181 (local address 127.14.14.190:36539)
0928 22:57:30.440873 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:30.440875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:30.441038 (+ 163us) server_negotiation.cc:407] Connection header received
0928 22:57:30.441630 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:30.441631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:30.441644 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:30.441711 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:30.442352 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.442831 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.443913 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.444122 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.444180 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.445051 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:30.445052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:30.445053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:30.445057 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:30.445077 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:30.445080 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:30.445119 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:30.445261 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 22:57:30.445280 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:57:30.445731 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.440719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:36539 (local address 127.14.14.131:53181)
0928 22:57:30.440910 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:30.440951 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:30.441503 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:30.441768 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:30.441772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:30.441786 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:30.442281 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.442285 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.442947 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.442948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.443823 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.443825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.444224 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.444225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.444344 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.444927 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:57:30.444927 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:30.444955 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:57:30.444977 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:30.445177 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:30.445182 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:30.445185 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:57:30.445263 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:57:30.445347 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260928 22:57:30.446272 32367 heartbeater.cc:347] Connected to a master server at 127.14.14.190:36539
I20260928 22:57:30.446370 32367 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:30.446599 32367 heartbeater.cc:511] Master 127.14.14.190:36539 requested a full tablet report, sending...
I20260928 22:57:30.447014 31917 ts_manager.cc:194] Registered new tserver with Master: 080e0d717c56460b9bd6da23205db1f5 (127.14.14.131:35101)
I20260928 22:57:30.447397 31917 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.14.131:53181
I20260928 22:57:30.447916 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 32239
I20260928 22:57:30.448017 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:57:30.448225 14394 external_mini_cluster.cc:1428] Setting key c4a0b42a5296b8519f49f684c8e02f8d
I20260928 22:57:30.448971 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:57:30.453253 31957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.450358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.14.14.190:36539)
0928 22:57:30.450394 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:30.450396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:30.450540 (+ 144us) server_negotiation.cc:407] Connection header received
0928 22:57:30.450640 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:30.450641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:30.450650 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:30.450706 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:30.450874 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.451350 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.452121 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.452303 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.452371 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.452992 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:30.452993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:30.452993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:30.452998 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:30.453013 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:30.453016 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:30.453092 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:30.453156 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 22:57:30.453175 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:57:30.454838 31917 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:57:30.456396 31917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:57:30.463095 32379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.459000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.14.14.131:35101)
0928 22:57:30.459214 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:30.459218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:30.459658 (+ 440us) server_negotiation.cc:407] Connection header received
0928 22:57:30.459717 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:30.459719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:30.459730 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:30.459813 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:30.459911 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.460587 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.462180 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.462680 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.462779 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.462869 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:30.462965 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 22:57:30.463007 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":110,"threads_started":1}
I20260928 22:57:30.463095 32376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.458876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:35101 (local address 127.0.0.1:53976)
0928 22:57:30.459601 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:30.459610 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:30.459656 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:30.459812 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:30.459813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:30.459815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:30.459857 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.459859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.460670 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.460671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.462076 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.462079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.462783 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.462784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.462863 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.462876 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 22:57:30.462951 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 22:57:30.463007 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":48,"thread_start_us":60,"threads_started":1}
I20260928 22:57:30.463312 32378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.458657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:46137 (local address 127.0.0.1:43204)
0928 22:57:30.459019 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:30.459031 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:30.459097 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:30.459303 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:30.459307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:30.459309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:30.459383 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.459385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.460721 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.460722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.462084 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.462085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.463069 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.463069 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.463127 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.463130 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:57:30.463177 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 22:57:30.463222 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":95,"thread_start_us":97,"threads_started":1}
I20260928 22:57:30.463382 32375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.458657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:34863 (local address 127.0.0.1:35324)
0928 22:57:30.458774 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:30.458811 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:30.458877 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:30.459469 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:30.459470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:30.459472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:30.459504 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.459506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.460874 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.460875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.461892 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.461895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.462896 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.462897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.462963 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.462967 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:57:30.463026 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:57:30.463070 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260928 22:57:30.463634 32377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.458748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.14.14.129:46137)
0928 22:57:30.458921 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:30.458926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:30.459104 (+ 178us) server_negotiation.cc:407] Connection header received
0928 22:57:30.459178 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:30.459181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:30.459198 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:30.459288 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:30.459471 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.460652 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.462165 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.462893 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.462955 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.463037 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:30.463465 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 22:57:30.463536 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260928 22:57:30.463883 32301 tablet_service.cc:1467] Processing CreateTablet for tablet bea96326be43452c8ec897983f3b883f (DEFAULT_TABLE table=test-table [id=3a3662c8d8894d8e9ecf1921bd7a734b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:30.464038 32380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.459036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35324 (local address 127.14.14.130:34863)
0928 22:57:30.459225 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:30.459235 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:30.459243 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:57:30.459295 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:30.459297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:30.459312 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:30.459401 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:30.459737 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.460787 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.461972 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.462692 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.462767 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.462857 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:30.463138 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 22:57:30.463955 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
I20260928 22:57:30.464275 32301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea96326be43452c8ec897983f3b883f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:30.464267 32039 tablet_service.cc:1467] Processing CreateTablet for tablet bea96326be43452c8ec897983f3b883f (DEFAULT_TABLE table=test-table [id=3a3662c8d8894d8e9ecf1921bd7a734b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:30.464583 32039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea96326be43452c8ec897983f3b883f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:30.464624 32170 tablet_service.cc:1467] Processing CreateTablet for tablet bea96326be43452c8ec897983f3b883f (DEFAULT_TABLE table=test-table [id=3a3662c8d8894d8e9ecf1921bd7a734b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:30.464960 32170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea96326be43452c8ec897983f3b883f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:30.467706 32381 tablet_bootstrap.cc:492] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512: Bootstrap starting.
I20260928 22:57:30.468063 32382 tablet_bootstrap.cc:492] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e: Bootstrap starting.
I20260928 22:57:30.468065 32383 tablet_bootstrap.cc:492] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5: Bootstrap starting.
I20260928 22:57:30.468454 32381 tablet_bootstrap.cc:654] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:30.468736 32383 tablet_bootstrap.cc:654] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:30.468735 32382 tablet_bootstrap.cc:654] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:30.468761 32381 log.cc:824] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:30.469069 32383 log.cc:824] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:30.469080 32382 log.cc:824] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:30.469576 32381 tablet_bootstrap.cc:492] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512: No bootstrap required, opened a new log
I20260928 22:57:30.469672 32381 ts_tablet_manager.cc:1397] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:57:30.469895 32382 tablet_bootstrap.cc:492] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e: No bootstrap required, opened a new log
I20260928 22:57:30.469969 32382 ts_tablet_manager.cc:1397] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:57:30.470180 32383 tablet_bootstrap.cc:492] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5: No bootstrap required, opened a new log
I20260928 22:57:30.470279 32383 ts_tablet_manager.cc:1397] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:57:30.471832 32381 raft_consensus.cc:348] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47ca03c3a09479ea84d18f7a283ca8e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46137 } } peers { permanent_uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34863 } } peers { permanent_uuid: "080e0d717c56460b9bd6da23205db1f5" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 35101 } }
I20260928 22:57:30.472002 32381 raft_consensus.cc:374] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:30.472047 32381 raft_consensus.cc:729] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52cc2c3cf9e4018afaf6a6bd4da3512, State: Initialized, Role: FOLLOWER
I20260928 22:57:30.472184 32381 consensus_queue.cc:260] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47ca03c3a09479ea84d18f7a283ca8e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46137 } } peers { permanent_uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34863 } } peers { permanent_uuid: "080e0d717c56460b9bd6da23205db1f5" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 35101 } }
I20260928 22:57:30.472483 32236 heartbeater.cc:503] Master 127.14.14.190:36539 was elected leader, sending a full tablet report...
I20260928 22:57:30.472736 32381 ts_tablet_manager.cc:1428] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:57:30.472725 32382 raft_consensus.cc:348] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47ca03c3a09479ea84d18f7a283ca8e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46137 } } peers { permanent_uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34863 } } peers { permanent_uuid: "080e0d717c56460b9bd6da23205db1f5" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 35101 } }
I20260928 22:57:30.472882 32382 raft_consensus.cc:374] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:30.472937 32382 raft_consensus.cc:729] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47ca03c3a09479ea84d18f7a283ca8e, State: Initialized, Role: FOLLOWER
I20260928 22:57:30.472920 32383 raft_consensus.cc:348] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47ca03c3a09479ea84d18f7a283ca8e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46137 } } peers { permanent_uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34863 } } peers { permanent_uuid: "080e0d717c56460b9bd6da23205db1f5" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 35101 } }
I20260928 22:57:30.473057 32383 raft_consensus.cc:374] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:30.473096 32383 raft_consensus.cc:729] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080e0d717c56460b9bd6da23205db1f5, State: Initialized, Role: FOLLOWER
I20260928 22:57:30.473058 32382 consensus_queue.cc:260] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47ca03c3a09479ea84d18f7a283ca8e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46137 } } peers { permanent_uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34863 } } peers { permanent_uuid: "080e0d717c56460b9bd6da23205db1f5" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 35101 } }
I20260928 22:57:30.473204 32383 consensus_queue.cc:260] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47ca03c3a09479ea84d18f7a283ca8e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46137 } } peers { permanent_uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34863 } } peers { permanent_uuid: "080e0d717c56460b9bd6da23205db1f5" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 35101 } }
I20260928 22:57:30.473330 32382 ts_tablet_manager.cc:1428] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 22:57:30.473457 32105 heartbeater.cc:503] Master 127.14.14.190:36539 was elected leader, sending a full tablet report...
I20260928 22:57:30.473451 32383 ts_tablet_manager.cc:1428] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:57:30.473603 32367 heartbeater.cc:503] Master 127.14.14.190:36539 was elected leader, sending a full tablet report...
W20260928 22:57:30.570452 32237 tablet.cc:2369] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:57:30.679330 32106 tablet.cc:2369] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:57:30.682000 32387 raft_consensus.cc:482] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:57:30.682126 32387 raft_consensus.cc:504] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47ca03c3a09479ea84d18f7a283ca8e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46137 } } peers { permanent_uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34863 } } peers { permanent_uuid: "080e0d717c56460b9bd6da23205db1f5" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 35101 } }
I20260928 22:57:30.682514 32387 leader_election.cc:288] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c47ca03c3a09479ea84d18f7a283ca8e (127.14.14.129:46137), 080e0d717c56460b9bd6da23205db1f5 (127.14.14.131:35101)
I20260928 22:57:30.685637 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.682831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:46137 (local address 127.14.14.130:51793)
0928 22:57:30.682904 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:30.682929 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:30.682998 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:30.683170 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:30.683171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:30.683173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:30.683266 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.683267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.683932 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.683933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.684793 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.684795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.685362 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.685363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.685437 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.685440 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:57:30.685504 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 22:57:30.685546 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 22:57:30.685799 32377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.682898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:51793 (local address 127.14.14.129:46137)
0928 22:57:30.682976 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:30.682978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:30.682990 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:57:30.683090 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:30.683092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:30.683106 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:30.683168 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:30.683339 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.683844 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.684904 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.685285 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.685359 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.685367 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:30.685509 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 22:57:30.685546 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:57:30.685930 32379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.683066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:52703 (local address 127.14.14.131:35101)
0928 22:57:30.683128 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:30.683130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:30.683262 (+ 132us) server_negotiation.cc:407] Connection header received
0928 22:57:30.683333 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:30.683334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:30.683343 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:30.683416 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:30.683566 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.684104 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.685011 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.685496 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.685608 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.685616 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:30.685805 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 22:57:30.685852 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:57:30.686086 32059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea96326be43452c8ec897983f3b883f" candidate_uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47ca03c3a09479ea84d18f7a283ca8e" is_pre_election: true
I20260928 22:57:30.686156 32390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.682987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:35101 (local address 127.14.14.130:52703)
0928 22:57:30.683180 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:30.683188 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:30.683275 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:30.683421 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:30.683421 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:30.683423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:30.683493 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.683495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.684170 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.684171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.684920 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:30.684922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.685607 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.685608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:30.685705 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.685709 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:57:30.686045 (+ 336us) client_negotiation.cc:240] Negotiation successful
0928 22:57:30.686084 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":115,"threads_started":1}
I20260928 22:57:30.686254 32059 raft_consensus.cc:2454] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b52cc2c3cf9e4018afaf6a6bd4da3512 in term 0.
I20260928 22:57:30.686474 32321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea96326be43452c8ec897983f3b883f" candidate_uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "080e0d717c56460b9bd6da23205db1f5" is_pre_election: true
I20260928 22:57:30.686522 32125 leader_election.cc:302] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52cc2c3cf9e4018afaf6a6bd4da3512, c47ca03c3a09479ea84d18f7a283ca8e; no voters:
I20260928 22:57:30.686619 32321 raft_consensus.cc:2454] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b52cc2c3cf9e4018afaf6a6bd4da3512 in term 0.
I20260928 22:57:30.686647 32387 raft_consensus.cc:2781] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:57:30.686705 32387 raft_consensus.cc:482] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:57:30.686729 32387 raft_consensus.cc:3037] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:30.687466 32387 raft_consensus.cc:504] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47ca03c3a09479ea84d18f7a283ca8e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46137 } } peers { permanent_uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34863 } } peers { permanent_uuid: "080e0d717c56460b9bd6da23205db1f5" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 35101 } }
I20260928 22:57:30.687598 32387 leader_election.cc:288] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [CANDIDATE]: Term 1 election: Requested vote from peers c47ca03c3a09479ea84d18f7a283ca8e (127.14.14.129:46137), 080e0d717c56460b9bd6da23205db1f5 (127.14.14.131:35101)
I20260928 22:57:30.687762 32321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea96326be43452c8ec897983f3b883f" candidate_uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "080e0d717c56460b9bd6da23205db1f5"
I20260928 22:57:30.687783 32059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea96326be43452c8ec897983f3b883f" candidate_uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47ca03c3a09479ea84d18f7a283ca8e"
I20260928 22:57:30.687855 32059 raft_consensus.cc:3037] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:30.687865 32321 raft_consensus.cc:3037] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:30.688586 32059 raft_consensus.cc:2454] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52cc2c3cf9e4018afaf6a6bd4da3512 in term 1.
I20260928 22:57:30.688747 32321 raft_consensus.cc:2454] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52cc2c3cf9e4018afaf6a6bd4da3512 in term 1.
I20260928 22:57:30.688759 32125 leader_election.cc:302] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52cc2c3cf9e4018afaf6a6bd4da3512, c47ca03c3a09479ea84d18f7a283ca8e; no voters:
I20260928 22:57:30.688834 32387 raft_consensus.cc:2781] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:30.689002 32387 raft_consensus.cc:686] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [term 1 LEADER]: Becoming Leader. State: Replica: b52cc2c3cf9e4018afaf6a6bd4da3512, State: Running, Role: LEADER
I20260928 22:57:30.689102 32387 consensus_queue.cc:237] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47ca03c3a09479ea84d18f7a283ca8e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46137 } } peers { permanent_uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34863 } } peers { permanent_uuid: "080e0d717c56460b9bd6da23205db1f5" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 35101 } }
I20260928 22:57:30.689895 31916 catalog_manager.cc:5495] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 reported cstate change: term changed from 0 to 1, leader changed from <none> to b52cc2c3cf9e4018afaf6a6bd4da3512 (127.14.14.130). New cstate: current_term: 1 leader_uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47ca03c3a09479ea84d18f7a283ca8e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b52cc2c3cf9e4018afaf6a6bd4da3512" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "080e0d717c56460b9bd6da23205db1f5" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 35101 } health_report { overall_health: UNKNOWN } } }
W20260928 22:57:30.690606 32368 tablet.cc:2369] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:57:30.750465 32380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.747827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.14.14.130:34863)
0928 22:57:30.747891 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:30.747893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:30.747954 (+ 61us) server_negotiation.cc:407] Connection header received
0928 22:57:30.747998 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:30.747999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:30.748018 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:30.748101 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:57:30.748237 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.748748 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.749537 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:30.749820 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:30.749893 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:30.750040 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:57:30.750177 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:57:30.750265 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:30.750320 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 22:57:30.750341 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:57:30.752465 32321 raft_consensus.cc:1261] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5 [term 1 FOLLOWER]: Refusing update from remote peer b52cc2c3cf9e4018afaf6a6bd4da3512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:57:30.752527 32059 raft_consensus.cc:1261] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e [term 1 FOLLOWER]: Refusing update from remote peer b52cc2c3cf9e4018afaf6a6bd4da3512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:57:30.752856 32387 consensus_queue.cc:1035] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c47ca03c3a09479ea84d18f7a283ca8e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 46137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:57:30.753001 32391 consensus_queue.cc:1035] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "080e0d717c56460b9bd6da23205db1f5" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:57:30.755839 32395 mvcc.cc:205] Tried to move back new op lower bound from 7334446083078766592 to 7334446082823237632. Current Snapshot: MvccSnapshot[applied={T|T < 7334446083078766592}]
I20260928 22:57:30.757496 31916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:57:30.757622 31916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:57:30.758324 31916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bca998ed4bfc47e29e7d82513e301a31: Sending DeleteTablet for 3 replicas of tablet bea96326be43452c8ec897983f3b883f
I20260928 22:57:30.758631 32039 tablet_service.cc:1514] Processing DeleteTablet for tablet bea96326be43452c8ec897983f3b883f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:30 UTC) from {username='slave'} at 127.0.0.1:43204
I20260928 22:57:30.758658 32170 tablet_service.cc:1514] Processing DeleteTablet for tablet bea96326be43452c8ec897983f3b883f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:30 UTC) from {username='slave'} at 127.0.0.1:35324
I20260928 22:57:30.758682 32301 tablet_service.cc:1514] Processing DeleteTablet for tablet bea96326be43452c8ec897983f3b883f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:30 UTC) from {username='slave'} at 127.0.0.1:53976
I20260928 22:57:30.758872 32404 tablet_replica.cc:330] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e: stopping tablet replica
I20260928 22:57:30.758993 32404 raft_consensus.cc:2229] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:57:30.759123 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 31977
I20260928 22:57:30.759029 32406 tablet_replica.cc:330] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512: stopping tablet replica
I20260928 22:57:30.759152 32404 raft_consensus.cc:2258] T bea96326be43452c8ec897983f3b883f P c47ca03c3a09479ea84d18f7a283ca8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:57:30.759361 32406 raft_consensus.cc:2229] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:57:30.759451 32406 raft_consensus.cc:2258] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:57:30.759868 32406 ts_tablet_manager.cc:1905] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:57:30.761055 32405 tablet_replica.cc:330] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5: stopping tablet replica
I20260928 22:57:30.761169 32405 raft_consensus.cc:2229] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:57:30.761269 32405 raft_consensus.cc:2258] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:57:30.761685 32406 ts_tablet_manager.cc:1918] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:57:30.761741 32405 ts_tablet_manager.cc:1905] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:57:30.761843 32406 log.cc:1196] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/bea96326be43452c8ec897983f3b883f
I20260928 22:57:30.762181 32406 ts_tablet_manager.cc:1939] T bea96326be43452c8ec897983f3b883f P b52cc2c3cf9e4018afaf6a6bd4da3512: Deleting consensus metadata
I20260928 22:57:30.762632 31904 catalog_manager.cc:4841] TS b52cc2c3cf9e4018afaf6a6bd4da3512 (127.14.14.130:34863): tablet bea96326be43452c8ec897983f3b883f (table test-table [id=3a3662c8d8894d8e9ecf1921bd7a734b]) successfully deleted
I20260928 22:57:30.763339 32405 ts_tablet_manager.cc:1918] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:57:30.763422 32405 log.cc:1196] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/bea96326be43452c8ec897983f3b883f
I20260928 22:57:30.763653 32405 ts_tablet_manager.cc:1939] T bea96326be43452c8ec897983f3b883f P 080e0d717c56460b9bd6da23205db1f5: Deleting consensus metadata
I20260928 22:57:30.764025 31903 catalog_manager.cc:4841] TS 080e0d717c56460b9bd6da23205db1f5 (127.14.14.131:35101): tablet bea96326be43452c8ec897983f3b883f (table test-table [id=3a3662c8d8894d8e9ecf1921bd7a734b]) successfully deleted
W20260928 22:57:30.766383 31903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:46137 (error 108)
I20260928 22:57:30.766572 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 32108
I20260928 22:57:30.767613 32378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:30.767485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:46137 (local address 127.0.0.1:43208)
0928 22:57:30.767536 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:30.767577 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:46137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:57:30.767774 31903 catalog_manager.cc:4584] TS c47ca03c3a09479ea84d18f7a283ca8e (127.14.14.129:46137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bea96326be43452c8ec897983f3b883f: Network error: Client connection negotiation failed: client connection to 127.14.14.129:46137: connect: Connection refused (error 111)
I20260928 22:57:30.771585 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 32239
I20260928 22:57:30.776160 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 31886
2026-09-28T22:57:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (909 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:30 dist-test-slave-kzfr krb5kdc[32410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:30 dist-test-slave-kzfr krb5kdc[32410](info): set up 2 sockets
Sep 28 22:57:30 dist-test-slave-kzfr krb5kdc[32410](info): commencing operation
krb5kdc: starting...
W20260928 22:57:31.854007 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:31 dist-test-slave-kzfr krb5kdc[32410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:57:31.972990 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:36535
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:36535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:32.053817 32423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:32.054037 32423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:32.054060 32423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:32.055189 32423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:32.055228 32423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:32.055243 32423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:32.055256 32423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:32.055269 32423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:32.056689 32423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:36535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:36535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:32.056881 32423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:32.057049 32423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:32.062157 32429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:32.062157 32431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:32.062186 32428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:32.062450 32423 server_base.cc:1034] running on GCE node
I20260928 22:57:32.062666 32423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:32.062906 32423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:32.064040 32423 hybrid_clock.cc:648] HybridClock initialized: now 1790636252064029 us; error 28 us; skew 500 ppm
Sep 28 22:57:32 dist-test-slave-kzfr krb5kdc[32410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636252, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:32.070297 32423 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:57:32.070788 32423 webserver.cc:460] Webserver started at http://127.14.14.190:46287/ using document root <none> and password file <none>
I20260928 22:57:32.071015 32423 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:32.071060 32423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:32.071208 32423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:32.072278 32423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "6792f7f6f887416cad01743b393eb85b"
format_stamp: "Formatted at 2026-09-28 22:57:32 on dist-test-slave-kzfr"
server_key: "0317654fd6ffcf6b090ca76bf8890e6d"
server_key_iv: "551683f7c26d9d81ce73ed2dc051a391"
server_key_version: "encryptionkey@0"
I20260928 22:57:32.072629 32423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "6792f7f6f887416cad01743b393eb85b"
format_stamp: "Formatted at 2026-09-28 22:57:32 on dist-test-slave-kzfr"
server_key: "0317654fd6ffcf6b090ca76bf8890e6d"
server_key_iv: "551683f7c26d9d81ce73ed2dc051a391"
server_key_version: "encryptionkey@0"
I20260928 22:57:32.074023 32423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:57:32.074755 32438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:32.074911 32423 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:32.075014 32423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "6792f7f6f887416cad01743b393eb85b"
format_stamp: "Formatted at 2026-09-28 22:57:32 on dist-test-slave-kzfr"
server_key: "0317654fd6ffcf6b090ca76bf8890e6d"
server_key_iv: "551683f7c26d9d81ce73ed2dc051a391"
server_key_version: "encryptionkey@0"
I20260928 22:57:32.075086 32423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:32.091691 32423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:32.093734 32423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:32.093904 32423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:32.097929 32423 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:36535
I20260928 22:57:32.098028 32490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:36535 every 8 connection(s)
I20260928 22:57:32.098376 32423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:32.099258 32491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:32.101512 32491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b: Bootstrap starting.
I20260928 22:57:32.102128 32491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:32.102362 32491 log.cc:824] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:32.103009 32491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b: No bootstrap required, opened a new log
I20260928 22:57:32.104511 32491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6792f7f6f887416cad01743b393eb85b" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36535 } }
I20260928 22:57:32.104674 32491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:32.104699 32491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6792f7f6f887416cad01743b393eb85b, State: Initialized, Role: FOLLOWER
I20260928 22:57:32.104800 32491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6792f7f6f887416cad01743b393eb85b" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36535 } }
I20260928 22:57:32.104882 32491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:32.104918 32491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:32.104967 32491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:32.105489 32491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6792f7f6f887416cad01743b393eb85b" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36535 } }
I20260928 22:57:32.105640 32491 leader_election.cc:302] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6792f7f6f887416cad01743b393eb85b; no voters:
I20260928 22:57:32.105821 32491 leader_election.cc:288] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:32.105872 32494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:32.106052 32494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [term 1 LEADER]: Becoming Leader. State: Replica: 6792f7f6f887416cad01743b393eb85b, State: Running, Role: LEADER
I20260928 22:57:32.106076 32491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:32.106144 32494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6792f7f6f887416cad01743b393eb85b" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36535 } }
I20260928 22:57:32.106540 32496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6792f7f6f887416cad01743b393eb85b. Latest consensus state: current_term: 1 leader_uuid: "6792f7f6f887416cad01743b393eb85b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6792f7f6f887416cad01743b393eb85b" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36535 } } }
I20260928 22:57:32.106626 32496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:32.106659 32495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6792f7f6f887416cad01743b393eb85b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6792f7f6f887416cad01743b393eb85b" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36535 } } }
I20260928 22:57:32.106758 32495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:32.106860 32502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:32.107404 32502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:32.108064 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 32423
I20260928 22:57:32.108182 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:32.108412 14394 external_mini_cluster.cc:1428] Setting key 293d4f65fcd5e54123268d41d2a32447
I20260928 22:57:32.109140 32502 catalog_manager.cc:1347] Generated new cluster ID: 23a321729a744eb49a4a7fe7a265818f
I20260928 22:57:32.109193 32502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:57:32 dist-test-slave-kzfr krb5kdc[32410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:57:32.124365 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.108908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.14.14.190:36535)
0928 22:57:32.113726 (+ 4818us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:32.113735 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:32.113771 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:57:32.113900 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:32.113907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:32.113965 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:32.114145 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:32.114882 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.116313 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.117326 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.117968 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.118077 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.120819 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:32.120823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:32.120824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:32.120835 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:32.122753 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:32.123153 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:32.123155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:32.123156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:32.123255 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:32.123473 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:32.123474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:32.123474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:32.123765 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:32.123883 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:32.124020 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 22:57:32.124154 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4767,"thread_start_us":62,"threads_started":1}
I20260928 22:57:32.125890 32502 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:32.126387 32502 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:32.135478 32502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b: Generated new TSK 0
I20260928 22:57:32.135690 32502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:57:32.165385 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36535
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:32.271431 32517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:32.271656 32517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:32.271683 32517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:32.272830 32517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:32.272868 32517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:32.272895 32517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:57:32.274386 32517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:32.274595 32517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:32.274773 32517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:32.280025 32525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:32.280045 32523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:32.280143 32517 server_base.cc:1034] running on GCE node
W20260928 22:57:32.280109 32522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:32.280447 32517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:32.280699 32517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:32.281860 32517 hybrid_clock.cc:648] HybridClock initialized: now 1790636252281849 us; error 27 us; skew 500 ppm
Sep 28 22:57:32 dist-test-slave-kzfr krb5kdc[32410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636252, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:32.284721 32517 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:57:32.285127 32517 webserver.cc:460] Webserver started at http://127.14.14.129:35079/ using document root <none> and password file <none>
I20260928 22:57:32.285324 32517 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:32.285364 32517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:32.285454 32517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:32.286556 32517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "6c4ce280e1ba40d6b0aec2532c76880d"
format_stamp: "Formatted at 2026-09-28 22:57:32 on dist-test-slave-kzfr"
server_key: "ef2e66b54d9b562537b7284b7d2c380f"
server_key_iv: "8fdf9d5d4eae189f5ce34edf2900d7c4"
server_key_version: "encryptionkey@0"
I20260928 22:57:32.286861 32517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "6c4ce280e1ba40d6b0aec2532c76880d"
format_stamp: "Formatted at 2026-09-28 22:57:32 on dist-test-slave-kzfr"
server_key: "ef2e66b54d9b562537b7284b7d2c380f"
server_key_iv: "8fdf9d5d4eae189f5ce34edf2900d7c4"
server_key_version: "encryptionkey@0"
I20260928 22:57:32.288177 32517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:57:32.288913 32532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:32.289085 32517 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:32.289167 32517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "6c4ce280e1ba40d6b0aec2532c76880d"
format_stamp: "Formatted at 2026-09-28 22:57:32 on dist-test-slave-kzfr"
server_key: "ef2e66b54d9b562537b7284b7d2c380f"
server_key_iv: "8fdf9d5d4eae189f5ce34edf2900d7c4"
server_key_version: "encryptionkey@0"
I20260928 22:57:32.289222 32517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:32.311343 32517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:32.313459 32517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:32.313619 32517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:32.313829 32517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:32.314182 32517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:32.314219 32517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:32.314241 32517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:32.314292 32517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:32.320017 32517 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:35933
I20260928 22:57:32.320060 32645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:35933 every 8 connection(s)
I20260928 22:57:32.320401 32517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:57:32.321383 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 32517
I20260928 22:57:32.321501 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:57:32.321740 14394 external_mini_cluster.cc:1428] Setting key c5044c9f67b17c0f1d9d026157061225
Sep 28 22:57:32 dist-test-slave-kzfr krb5kdc[32410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636252, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
I20260928 22:57:32.330837 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.321183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:33403 (local address 127.14.14.190:36535)
0928 22:57:32.321263 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:32.321266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:32.321487 (+ 221us) server_negotiation.cc:407] Connection header received
0928 22:57:32.322124 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:32.322126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:32.322141 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:32.322194 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:32.323164 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.323972 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.324961 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.325309 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.325384 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.327991 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:32.327993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:32.327993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:32.328001 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:32.329351 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:32.329857 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:32.329857 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:32.329858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:32.329911 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:32.330176 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:32.330177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:32.330178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:32.330276 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:32.330325 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:32.330675 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 22:57:32.330749 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 22:57:32.330977 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.321145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:36535 (local address 127.14.14.129:33403)
0928 22:57:32.321341 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:32.321383 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:32.322019 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:32.322273 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:32.322277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:32.322857 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:32.323062 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.323067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.324069 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.324070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.324878 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.324880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.325398 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.325398 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.325524 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.326145 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:57:32.326153 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:32.327884 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:32.329435 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:32.329443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:32.329446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:32.329780 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:32.329974 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:32.329975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:32.329975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:32.330111 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:32.330337 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:32.330344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:32.330518 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 22:57:32.330634 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 22:57:32.330777 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":120,"threads_started":1}
I20260928 22:57:32.331570 32646 heartbeater.cc:347] Connected to a master server at 127.14.14.190:36535
I20260928 22:57:32.331704 32646 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:32.331898 32646 heartbeater.cc:511] Master 127.14.14.190:36535 requested a full tablet report, sending...
I20260928 22:57:32.332453 32455 ts_manager.cc:194] Registered new tserver with Master: 6c4ce280e1ba40d6b0aec2532c76880d (127.14.14.129:35933)
I20260928 22:57:32.333228 32455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:33403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:57:32.349498 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36535
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:32.432165 32651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:32.432364 32651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:32.432389 32651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:32.433449 32651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:32.433485 32651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:32.433511 32651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:57:32.435021 32651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:32.435212 32651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:32.435375 32651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:32.440222 32659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:32.440215 32657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:32.440215 32656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:32.440276 32651 server_base.cc:1034] running on GCE node
I20260928 22:57:32.440680 32651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:32.440932 32651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:32.442096 32651 hybrid_clock.cc:648] HybridClock initialized: now 1790636252442086 us; error 38 us; skew 500 ppm
Sep 28 22:57:32 dist-test-slave-kzfr krb5kdc[32410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636252, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:32.444715 32651 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:57:32.445186 32651 webserver.cc:460] Webserver started at http://127.14.14.130:39381/ using document root <none> and password file <none>
I20260928 22:57:32.445425 32651 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:32.445492 32651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:32.445664 32651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:32.446727 32651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "5f6e0bae30754890a9fd963b02068bed"
format_stamp: "Formatted at 2026-09-28 22:57:32 on dist-test-slave-kzfr"
server_key: "9b42f2a618e67e9ad1ddf2f9f8ec69b3"
server_key_iv: "9d4ba63ef05ff3fdeb05d4baa2f37ab6"
server_key_version: "encryptionkey@0"
I20260928 22:57:32.447096 32651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "5f6e0bae30754890a9fd963b02068bed"
format_stamp: "Formatted at 2026-09-28 22:57:32 on dist-test-slave-kzfr"
server_key: "9b42f2a618e67e9ad1ddf2f9f8ec69b3"
server_key_iv: "9d4ba63ef05ff3fdeb05d4baa2f37ab6"
server_key_version: "encryptionkey@0"
I20260928 22:57:32.448493 32651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:57:32.449223 32666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:32.449465 32651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:57:32.449571 32651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "5f6e0bae30754890a9fd963b02068bed"
format_stamp: "Formatted at 2026-09-28 22:57:32 on dist-test-slave-kzfr"
server_key: "9b42f2a618e67e9ad1ddf2f9f8ec69b3"
server_key_iv: "9d4ba63ef05ff3fdeb05d4baa2f37ab6"
server_key_version: "encryptionkey@0"
I20260928 22:57:32.449663 32651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:32.475302 32651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:32.477567 32651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:32.477823 32651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:32.478081 32651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:32.478457 32651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:32.478508 32651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:32.478552 32651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:32.478578 32651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:32.484503 32651 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:44131
I20260928 22:57:32.484545 311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:44131 every 8 connection(s)
I20260928 22:57:32.484923 32651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:57:32 dist-test-slave-kzfr krb5kdc[32410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636252, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:57:32.495026 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 32651
I20260928 22:57:32.495177 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:57:32.495340 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.485695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:46601 (local address 127.14.14.190:36535)
0928 22:57:32.485747 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:32.485751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:32.486014 (+ 263us) server_negotiation.cc:407] Connection header received
0928 22:57:32.486719 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:32.486721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:32.486736 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:32.486808 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:32.487763 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.488284 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.489291 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.489568 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.489730 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.491964 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:32.491966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:32.491966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:32.491976 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:32.493796 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:32.494226 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:32.494227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:32.494227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:32.494285 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:32.494590 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:32.494591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:32.494591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:32.494711 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:32.494778 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:32.495147 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 22:57:32.495242 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:57:32.495388 14394 external_mini_cluster.cc:1428] Setting key b168d88c32cc54b0fbf7d8d3d2c64399
I20260928 22:57:32.495592 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.485721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:36535 (local address 127.14.14.130:46601)
0928 22:57:32.485908 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:32.485946 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:32.486440 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:32.486881 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:32.486885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:32.487475 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:32.487672 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.487676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.488360 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.488361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.489184 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.489186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.489727 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.489728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.489835 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.490391 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:57:32.490399 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:32.491862 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:32.493884 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:32.493891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:32.493892 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:32.494129 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:32.494372 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:32.494373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:32.494373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:32.494504 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:32.494786 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:32.494790 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:32.494979 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 22:57:32.495070 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 22:57:32.495253 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":99,"threads_started":1}
I20260928 22:57:32.496163 312 heartbeater.cc:347] Connected to a master server at 127.14.14.190:36535
I20260928 22:57:32.496277 312 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:32.496469 312 heartbeater.cc:511] Master 127.14.14.190:36535 requested a full tablet report, sending...
I20260928 22:57:32.496899 32455 ts_manager.cc:194] Registered new tserver with Master: 5f6e0bae30754890a9fd963b02068bed (127.14.14.130:44131)
I20260928 22:57:32.497318 32455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:46601
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:57:32.520666 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36535
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:32.603598 317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:32.603790 317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:32.603816 317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:32.605010 317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:32.605048 317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:32.605074 317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:57:32.606709 317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:32.606937 317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:32.607178 317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:32.612430 323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:32.612584 322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:32.612497 317 server_base.cc:1034] running on GCE node
W20260928 22:57:32.612437 325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:32.612913 317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:32.613173 317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:32.614313 317 hybrid_clock.cc:648] HybridClock initialized: now 1790636252614280 us; error 45 us; skew 500 ppm
Sep 28 22:57:32 dist-test-slave-kzfr krb5kdc[32410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636252, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:32.616901 317 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:57:32.617352 317 webserver.cc:460] Webserver started at http://127.14.14.131:46171/ using document root <none> and password file <none>
I20260928 22:57:32.617676 317 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:32.617724 317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:32.617863 317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:32.618912 317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "f167c18243964d73bf852ccdaf7ae3ce"
format_stamp: "Formatted at 2026-09-28 22:57:32 on dist-test-slave-kzfr"
server_key: "cbb0bce25054c1ffc6513282df6420be"
server_key_iv: "1475bee3d4c95c2c3cc3d56b1b288993"
server_key_version: "encryptionkey@0"
I20260928 22:57:32.619247 317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "f167c18243964d73bf852ccdaf7ae3ce"
format_stamp: "Formatted at 2026-09-28 22:57:32 on dist-test-slave-kzfr"
server_key: "cbb0bce25054c1ffc6513282df6420be"
server_key_iv: "1475bee3d4c95c2c3cc3d56b1b288993"
server_key_version: "encryptionkey@0"
I20260928 22:57:32.620581 317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:57:32.621301 332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:32.621563 317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:57:32.621677 317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "f167c18243964d73bf852ccdaf7ae3ce"
format_stamp: "Formatted at 2026-09-28 22:57:32 on dist-test-slave-kzfr"
server_key: "cbb0bce25054c1ffc6513282df6420be"
server_key_iv: "1475bee3d4c95c2c3cc3d56b1b288993"
server_key_version: "encryptionkey@0"
I20260928 22:57:32.621734 317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:32.638146 317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:32.640120 317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:32.640235 317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:32.640416 317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:32.640667 317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:32.640702 317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:32.640723 317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:32.640736 317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:32.646492 317 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:44311
I20260928 22:57:32.646610 445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:44311 every 8 connection(s)
I20260928 22:57:32.646863 317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:57:32 dist-test-slave-kzfr krb5kdc[32410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636252, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:57:32.655570 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 317
I20260928 22:57:32.655666 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:57:32.655866 14394 external_mini_cluster.cc:1428] Setting key e19a96c87a7eebd5ec7b18a8f54e0a94
I20260928 22:57:32.656172 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.647553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:36769 (local address 127.14.14.190:36535)
0928 22:57:32.647613 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:32.647615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:32.647825 (+ 210us) server_negotiation.cc:407] Connection header received
0928 22:57:32.648323 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:32.648325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:32.648336 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:32.648398 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:32.649355 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.650018 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.651031 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.651210 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.651262 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.653450 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:32.653451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:32.653451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:32.653458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:32.654705 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:32.655063 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:32.655064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:32.655065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:32.655122 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:32.655350 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:32.655351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:32.655351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:32.655525 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:32.655582 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:32.655962 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 22:57:32.656050 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:57:32.656297 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.647545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:36535 (local address 127.14.14.131:36769)
0928 22:57:32.647710 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:32.647744 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:32.648199 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:32.648463 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:32.648466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:32.649073 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:32.649255 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.649260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.650095 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.650096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.650954 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.650956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.651275 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.651275 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.651372 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.651891 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:57:32.651898 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:32.653347 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:32.654771 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:32.654776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:32.654778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:32.654977 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:32.655186 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:32.655187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:32.655187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:32.655297 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:32.655596 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:32.655602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:32.655770 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 22:57:32.655890 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 22:57:32.656055 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260928 22:57:32.656739 446 heartbeater.cc:347] Connected to a master server at 127.14.14.190:36535
I20260928 22:57:32.656841 446 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:32.657035 446 heartbeater.cc:511] Master 127.14.14.190:36535 requested a full tablet report, sending...
I20260928 22:57:32.657454 32455 ts_manager.cc:194] Registered new tserver with Master: f167c18243964d73bf852ccdaf7ae3ce (127.14.14.131:44311)
I20260928 22:57:32.657954 32455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:36769
I20260928 22:57:32.667979 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:57:32.675694 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.669379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.14.14.190:36535)
0928 22:57:32.669431 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:32.669433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:32.669547 (+ 114us) server_negotiation.cc:407] Connection header received
0928 22:57:32.669638 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:32.669638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:32.669649 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:32.669703 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:32.670291 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.670782 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.671508 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.671770 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.671828 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.673114 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:32.673115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:32.673115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:32.673121 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:32.674601 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:32.674976 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:32.674977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:32.674977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:32.675028 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:32.675201 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:32.675202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:32.675202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:32.675322 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:32.675398 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:32.675585 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 22:57:32.675632 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:57:32.677491 32455 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:57:32.679471 32455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:57:32.686074 460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.682109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.14.14.130:44131)
0928 22:57:32.682516 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:32.682521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:32.682531 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:57:32.682567 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:32.682571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:32.682595 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:32.682670 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:32.682857 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.683639 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.685024 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.685678 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.685769 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.685830 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:32.685949 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 22:57:32.685989 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":86,"threads_started":1}
I20260928 22:57:32.686074 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.681935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:44131 (local address 127.0.0.1:53888)
0928 22:57:32.682276 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:32.682306 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:32.682372 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:32.682677 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:32.682679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:32.682682 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:32.682758 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.682760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.683733 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.683734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.684925 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.684928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.685771 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.685772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.685849 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.685862 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 22:57:32.685929 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 22:57:32.685989 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
I20260928 22:57:32.686447 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.682040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.14.14.129:35933)
0928 22:57:32.682218 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:32.682222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:32.682477 (+ 255us) server_negotiation.cc:407] Connection header received
0928 22:57:32.682526 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:32.682529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:32.682547 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:32.682622 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:32.682852 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.684012 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.685282 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.686022 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.686081 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.686144 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:32.686326 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 22:57:32.686373 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20260928 22:57:32.686240 461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.682127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:44311 (local address 127.0.0.1:60974)
0928 22:57:32.682280 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:32.682306 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:32.682372 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:32.682972 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:32.682973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:32.682976 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:32.683007 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.683009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.684178 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.684179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.685073 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.685076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.686012 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.686014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.686085 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.686089 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:57:32.686120 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 22:57:32.686167 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260928 22:57:32.686542 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.682227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60974 (local address 127.14.14.131:44311)
0928 22:57:32.682811 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:32.682815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:32.682825 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:57:32.682861 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:32.682863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:32.682877 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:32.682968 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:32.683070 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.684102 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.685159 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.685890 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.686313 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.686375 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:32.686400 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 22:57:32.686443 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":86,"threads_started":1}
I20260928 22:57:32.686393 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.681902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:35933 (local address 127.0.0.1:44982)
0928 22:57:32.682263 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:32.682306 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:32.682372 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:32.682710 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:32.682711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:32.682713 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:32.682766 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.682768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.684117 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.684118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.685207 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.685209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.686176 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.686177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.686235 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.686239 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:57:32.686297 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:57:32.686335 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":26,"thread_start_us":71,"threads_started":1}
I20260928 22:57:32.687197 380 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2c136dfc9a4dad9dd018ac25f74886 (DEFAULT_TABLE table=test-table [id=0d6d3744e9da4eaaabd5b107e4c79e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:32.687530 380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2c136dfc9a4dad9dd018ac25f74886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:32.690150 463 tablet_bootstrap.cc:492] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce: Bootstrap starting.
I20260928 22:57:32.690222 32714 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2c136dfc9a4dad9dd018ac25f74886 (DEFAULT_TABLE table=test-table [id=0d6d3744e9da4eaaabd5b107e4c79e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:32.690405 32580 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2c136dfc9a4dad9dd018ac25f74886 (DEFAULT_TABLE table=test-table [id=0d6d3744e9da4eaaabd5b107e4c79e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:32.690641 32580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2c136dfc9a4dad9dd018ac25f74886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:32.690649 32714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2c136dfc9a4dad9dd018ac25f74886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:32.690864 463 tablet_bootstrap.cc:654] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:32.691502 463 log.cc:824] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:32.692327 463 tablet_bootstrap.cc:492] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce: No bootstrap required, opened a new log
I20260928 22:57:32.692390 463 ts_tablet_manager.cc:1397] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:57:32.693729 465 tablet_bootstrap.cc:492] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d: Bootstrap starting.
I20260928 22:57:32.694106 466 tablet_bootstrap.cc:492] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed: Bootstrap starting.
I20260928 22:57:32.694443 465 tablet_bootstrap.cc:654] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:32.694535 463 raft_consensus.cc:348] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4ce280e1ba40d6b0aec2532c76880d" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35933 } } peers { permanent_uuid: "5f6e0bae30754890a9fd963b02068bed" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44131 } } peers { permanent_uuid: "f167c18243964d73bf852ccdaf7ae3ce" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 44311 } }
I20260928 22:57:32.694720 463 raft_consensus.cc:374] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:32.694764 463 raft_consensus.cc:729] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f167c18243964d73bf852ccdaf7ae3ce, State: Initialized, Role: FOLLOWER
I20260928 22:57:32.694831 466 tablet_bootstrap.cc:654] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:32.694880 463 consensus_queue.cc:260] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4ce280e1ba40d6b0aec2532c76880d" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35933 } } peers { permanent_uuid: "5f6e0bae30754890a9fd963b02068bed" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44131 } } peers { permanent_uuid: "f167c18243964d73bf852ccdaf7ae3ce" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 44311 } }
I20260928 22:57:32.695034 465 log.cc:824] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:32.695211 463 ts_tablet_manager.cc:1428] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:57:32.695225 446 heartbeater.cc:503] Master 127.14.14.190:36535 was elected leader, sending a full tablet report...
I20260928 22:57:32.695422 466 log.cc:824] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:32.695778 465 tablet_bootstrap.cc:492] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d: No bootstrap required, opened a new log
I20260928 22:57:32.695827 465 ts_tablet_manager.cc:1397] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:57:32.696170 466 tablet_bootstrap.cc:492] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed: No bootstrap required, opened a new log
I20260928 22:57:32.696231 466 ts_tablet_manager.cc:1397] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:57:32.697800 465 raft_consensus.cc:348] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4ce280e1ba40d6b0aec2532c76880d" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35933 } } peers { permanent_uuid: "5f6e0bae30754890a9fd963b02068bed" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44131 } } peers { permanent_uuid: "f167c18243964d73bf852ccdaf7ae3ce" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 44311 } }
I20260928 22:57:32.697983 465 raft_consensus.cc:374] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:32.698030 465 raft_consensus.cc:729] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4ce280e1ba40d6b0aec2532c76880d, State: Initialized, Role: FOLLOWER
I20260928 22:57:32.698153 465 consensus_queue.cc:260] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4ce280e1ba40d6b0aec2532c76880d" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35933 } } peers { permanent_uuid: "5f6e0bae30754890a9fd963b02068bed" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44131 } } peers { permanent_uuid: "f167c18243964d73bf852ccdaf7ae3ce" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 44311 } }
I20260928 22:57:32.698417 465 ts_tablet_manager.cc:1428] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:57:32.698438 32646 heartbeater.cc:503] Master 127.14.14.190:36535 was elected leader, sending a full tablet report...
I20260928 22:57:32.698347 466 raft_consensus.cc:348] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4ce280e1ba40d6b0aec2532c76880d" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35933 } } peers { permanent_uuid: "5f6e0bae30754890a9fd963b02068bed" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44131 } } peers { permanent_uuid: "f167c18243964d73bf852ccdaf7ae3ce" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 44311 } }
I20260928 22:57:32.698515 466 raft_consensus.cc:374] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:32.698560 466 raft_consensus.cc:729] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f6e0bae30754890a9fd963b02068bed, State: Initialized, Role: FOLLOWER
I20260928 22:57:32.698750 466 consensus_queue.cc:260] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4ce280e1ba40d6b0aec2532c76880d" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35933 } } peers { permanent_uuid: "5f6e0bae30754890a9fd963b02068bed" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44131 } } peers { permanent_uuid: "f167c18243964d73bf852ccdaf7ae3ce" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 44311 } }
I20260928 22:57:32.699038 466 ts_tablet_manager.cc:1428] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:57:32.699090 312 heartbeater.cc:503] Master 127.14.14.190:36535 was elected leader, sending a full tablet report...
W20260928 22:57:32.735599 313 tablet.cc:2369] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:57:32.821254 32647 tablet.cc:2369] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:57:32.873560 471 raft_consensus.cc:482] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:57:32.873759 471 raft_consensus.cc:504] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4ce280e1ba40d6b0aec2532c76880d" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35933 } } peers { permanent_uuid: "5f6e0bae30754890a9fd963b02068bed" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44131 } } peers { permanent_uuid: "f167c18243964d73bf852ccdaf7ae3ce" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 44311 } }
I20260928 22:57:32.874133 471 leader_election.cc:288] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c4ce280e1ba40d6b0aec2532c76880d (127.14.14.129:35933), f167c18243964d73bf852ccdaf7ae3ce (127.14.14.131:44311)
I20260928 22:57:32.877156 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.874349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:35933 (local address 127.14.14.130:60561)
0928 22:57:32.874483 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:32.874500 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:32.874569 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:32.874766 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:32.874768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:32.874770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:32.874846 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.874848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.875581 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.875582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.876356 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.876359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.876895 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.876896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.876975 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.876979 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:57:32.877042 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 22:57:32.877086 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":3}
I20260928 22:57:32.877168 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.874574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:60561 (local address 127.14.14.129:35933)
0928 22:57:32.874623 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:32.874625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:32.874633 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:57:32.874664 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:32.874665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:32.874674 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:32.874739 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:32.874942 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.875491 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.876473 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.876819 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.876872 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.876879 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:32.877062 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 22:57:32.877097 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:57:32.877568 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.874465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:44311 (local address 127.14.14.130:48819)
0928 22:57:32.874689 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:32.874697 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:32.874736 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:32.874904 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:32.874905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:32.874907 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:32.874980 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.874982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.875641 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.875641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.876357 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:32.876358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.877088 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.877089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:32.877157 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.877160 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:57:32.877449 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 22:57:32.877487 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":2,"thread_start_us":137,"threads_started":1}
I20260928 22:57:32.877517 32600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2c136dfc9a4dad9dd018ac25f74886" candidate_uuid: "5f6e0bae30754890a9fd963b02068bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4ce280e1ba40d6b0aec2532c76880d" is_pre_election: true
I20260928 22:57:32.877561 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.874636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:48819 (local address 127.14.14.131:44311)
0928 22:57:32.874707 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:32.874709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:32.874734 (+ 25us) server_negotiation.cc:407] Connection header received
0928 22:57:32.874799 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:32.874799 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:32.874810 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:32.874877 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:32.875050 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.875563 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.876575 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.877016 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.877080 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.877086 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:32.877477 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 22:57:32.877513 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:57:32.877705 32600 raft_consensus.cc:2454] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f6e0bae30754890a9fd963b02068bed in term 0.
I20260928 22:57:32.877939 32668 leader_election.cc:302] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f6e0bae30754890a9fd963b02068bed, 6c4ce280e1ba40d6b0aec2532c76880d; no voters:
I20260928 22:57:32.878072 400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2c136dfc9a4dad9dd018ac25f74886" candidate_uuid: "5f6e0bae30754890a9fd963b02068bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f167c18243964d73bf852ccdaf7ae3ce" is_pre_election: true
I20260928 22:57:32.878106 471 raft_consensus.cc:2781] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:57:32.878160 471 raft_consensus.cc:482] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:57:32.878187 471 raft_consensus.cc:3037] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:32.878221 400 raft_consensus.cc:2454] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f6e0bae30754890a9fd963b02068bed in term 0.
I20260928 22:57:32.878995 471 raft_consensus.cc:504] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4ce280e1ba40d6b0aec2532c76880d" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35933 } } peers { permanent_uuid: "5f6e0bae30754890a9fd963b02068bed" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44131 } } peers { permanent_uuid: "f167c18243964d73bf852ccdaf7ae3ce" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 44311 } }
I20260928 22:57:32.879143 471 leader_election.cc:288] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [CANDIDATE]: Term 1 election: Requested vote from peers 6c4ce280e1ba40d6b0aec2532c76880d (127.14.14.129:35933), f167c18243964d73bf852ccdaf7ae3ce (127.14.14.131:44311)
I20260928 22:57:32.879309 32600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2c136dfc9a4dad9dd018ac25f74886" candidate_uuid: "5f6e0bae30754890a9fd963b02068bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4ce280e1ba40d6b0aec2532c76880d"
I20260928 22:57:32.879333 400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2c136dfc9a4dad9dd018ac25f74886" candidate_uuid: "5f6e0bae30754890a9fd963b02068bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f167c18243964d73bf852ccdaf7ae3ce"
I20260928 22:57:32.879383 32600 raft_consensus.cc:3037] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:32.879419 400 raft_consensus.cc:3037] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:32.880040 32600 raft_consensus.cc:2454] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f6e0bae30754890a9fd963b02068bed in term 1.
I20260928 22:57:32.880198 32668 leader_election.cc:302] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f6e0bae30754890a9fd963b02068bed, 6c4ce280e1ba40d6b0aec2532c76880d; no voters:
I20260928 22:57:32.880215 400 raft_consensus.cc:2454] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f6e0bae30754890a9fd963b02068bed in term 1.
I20260928 22:57:32.880301 471 raft_consensus.cc:2781] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:32.880462 471 raft_consensus.cc:686] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [term 1 LEADER]: Becoming Leader. State: Replica: 5f6e0bae30754890a9fd963b02068bed, State: Running, Role: LEADER
I20260928 22:57:32.880669 471 consensus_queue.cc:237] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4ce280e1ba40d6b0aec2532c76880d" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35933 } } peers { permanent_uuid: "5f6e0bae30754890a9fd963b02068bed" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44131 } } peers { permanent_uuid: "f167c18243964d73bf852ccdaf7ae3ce" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 44311 } }
I20260928 22:57:32.881528 32455 catalog_manager.cc:5495] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f6e0bae30754890a9fd963b02068bed (127.14.14.130). New cstate: current_term: 1 leader_uuid: "5f6e0bae30754890a9fd963b02068bed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c4ce280e1ba40d6b0aec2532c76880d" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f6e0bae30754890a9fd963b02068bed" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 44131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f167c18243964d73bf852ccdaf7ae3ce" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 44311 } health_report { overall_health: UNKNOWN } } }
W20260928 22:57:32.897414 447 tablet.cc:2369] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:57:32.912777 460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.910091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.14.14.130:44131)
0928 22:57:32.910220 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:32.910222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:32.910239 (+ 17us) server_negotiation.cc:407] Connection header received
0928 22:57:32.910280 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:32.910281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:32.910299 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:32.910360 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:57:32.910500 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.910962 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.911860 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:32.912057 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:32.912116 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:32.912331 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:57:32.912511 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:57:32.912613 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:32.912690 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 22:57:32.912700 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 22:57:32.915092 400 raft_consensus.cc:1261] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce [term 1 FOLLOWER]: Refusing update from remote peer 5f6e0bae30754890a9fd963b02068bed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:57:32.915108 32600 raft_consensus.cc:1261] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d [term 1 FOLLOWER]: Refusing update from remote peer 5f6e0bae30754890a9fd963b02068bed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:57:32.915495 471 consensus_queue.cc:1035] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c4ce280e1ba40d6b0aec2532c76880d" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:57:32.915740 473 consensus_queue.cc:1035] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "f167c18243964d73bf852ccdaf7ae3ce" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 44311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:57:32.917991 476 mvcc.cc:205] Tried to move back new op lower bound from 7334446091937062912 to 7334446091799818240. Current Snapshot: MvccSnapshot[applied={T|T < 7334446091937062912}]
I20260928 22:57:32.918210 477 mvcc.cc:205] Tried to move back new op lower bound from 7334446091937062912 to 7334446091799818240. Current Snapshot: MvccSnapshot[applied={T|T < 7334446091937062912}]
I20260928 22:57:32.918737 478 mvcc.cc:205] Tried to move back new op lower bound from 7334446091937062912 to 7334446091799818240. Current Snapshot: MvccSnapshot[applied={T|T < 7334446091937062912}]
I20260928 22:57:32.920708 32455 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:57:32.920807 32455 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:57:32.921670 32455 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6792f7f6f887416cad01743b393eb85b: Sending DeleteTablet for 3 replicas of tablet 5d2c136dfc9a4dad9dd018ac25f74886
I20260928 22:57:32.922048 32714 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2c136dfc9a4dad9dd018ac25f74886 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:32 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:53888
I20260928 22:57:32.922063 32580 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2c136dfc9a4dad9dd018ac25f74886 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:32 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:44982
I20260928 22:57:32.922111 380 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2c136dfc9a4dad9dd018ac25f74886 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:32 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:60974
I20260928 22:57:32.922290 486 tablet_replica.cc:330] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed: stopping tablet replica
I20260928 22:57:32.922329 487 tablet_replica.cc:330] T 5d2c136dfc9a4dad9dd018ac25f74886 P 6c4ce280e1ba40d6b0aec2532c76880d: stopping tablet replica
I20260928 22:57:32.922401 486 raft_consensus.cc:2229] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:57:32.922461 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 32517
I20260928 22:57:32.923050 486 raft_consensus.cc:2258] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:57:32.923558 486 ts_tablet_manager.cc:1905] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:57:32.923801 488 tablet_replica.cc:330] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce: stopping tablet replica
I20260928 22:57:32.923907 488 raft_consensus.cc:2229] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:57:32.924018 488 raft_consensus.cc:2258] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:57:32.924471 488 ts_tablet_manager.cc:1905] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:57:32.925648 486 ts_tablet_manager.cc:1918] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:57:32.925729 486 log.cc:1196] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/5d2c136dfc9a4dad9dd018ac25f74886
I20260928 22:57:32.926051 486 ts_tablet_manager.cc:1939] T 5d2c136dfc9a4dad9dd018ac25f74886 P 5f6e0bae30754890a9fd963b02068bed: Deleting consensus metadata
I20260928 22:57:32.926378 488 ts_tablet_manager.cc:1918] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:57:32.926474 488 log.cc:1196] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/5d2c136dfc9a4dad9dd018ac25f74886
I20260928 22:57:32.926517 32441 catalog_manager.cc:4841] TS 5f6e0bae30754890a9fd963b02068bed (127.14.14.130:44131): tablet 5d2c136dfc9a4dad9dd018ac25f74886 (table test-table [id=0d6d3744e9da4eaaabd5b107e4c79e4b]) successfully deleted
I20260928 22:57:32.926779 488 ts_tablet_manager.cc:1939] T 5d2c136dfc9a4dad9dd018ac25f74886 P f167c18243964d73bf852ccdaf7ae3ce: Deleting consensus metadata
I20260928 22:57:32.927455 32440 catalog_manager.cc:4841] TS f167c18243964d73bf852ccdaf7ae3ce (127.14.14.131:44311): tablet 5d2c136dfc9a4dad9dd018ac25f74886 (table test-table [id=0d6d3744e9da4eaaabd5b107e4c79e4b]) successfully deleted
W20260928 22:57:32.928622 32440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:35933 (error 108)
I20260928 22:57:32.928848 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 32651
I20260928 22:57:32.930146 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:32.929729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:35933 (local address 127.0.0.1:44994)
0928 22:57:32.930073 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:32.930116 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:35933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321}
W20260928 22:57:32.930315 32440 catalog_manager.cc:4584] TS 6c4ce280e1ba40d6b0aec2532c76880d (127.14.14.129:35933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d2c136dfc9a4dad9dd018ac25f74886: Network error: Client connection negotiation failed: client connection to 127.14.14.129:35933: connect: Connection refused (error 111)
I20260928 22:57:32.934793 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 317
I20260928 22:57:32.939435 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 32423
2026-09-28T22:57:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2165 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T22:57:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:32Z Disabled control of system clock
I20260928 22:57:32.967671 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44621
--webserver_interface=127.14.14.190
--webserver_port=0
--builtin_ntp_servers=127.14.14.148:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:44621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:57:33.049352 493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:33.049608 493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:33.049633 493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:33.050838 493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:33.050884 493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:33.050905 493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:33.050947 493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:33.050974 493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:33.052418 493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:44621
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:33.052690 493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:33.052995 493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:33.058015 498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:33.057996 501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:33.057996 499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:33.058243 493 server_base.cc:1034] running on GCE node
I20260928 22:57:33.058476 493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:33.058765 493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:33.059937 493 hybrid_clock.cc:648] HybridClock initialized: now 1790636253059913 us; error 35 us; skew 500 ppm
I20260928 22:57:33.061252 493 webserver.cc:460] Webserver started at http://127.14.14.190:41779/ using document root <none> and password file <none>
I20260928 22:57:33.061655 493 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:33.061712 493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:33.061862 493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:33.062954 493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "962e1b7171f143ffa08fd12ddbba72b1"
format_stamp: "Formatted at 2026-09-28 22:57:33 on dist-test-slave-kzfr"
server_key: "7b5cbd5de9e08a5d4b7e92e7e5fc87f7"
server_key_iv: "ebf38a0db866824011792ce97e3b877f"
server_key_version: "encryptionkey@0"
I20260928 22:57:33.063345 493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "962e1b7171f143ffa08fd12ddbba72b1"
format_stamp: "Formatted at 2026-09-28 22:57:33 on dist-test-slave-kzfr"
server_key: "7b5cbd5de9e08a5d4b7e92e7e5fc87f7"
server_key_iv: "ebf38a0db866824011792ce97e3b877f"
server_key_version: "encryptionkey@0"
I20260928 22:57:33.064714 493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:57:33.065544 507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:33.065814 493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:33.065920 493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "962e1b7171f143ffa08fd12ddbba72b1"
format_stamp: "Formatted at 2026-09-28 22:57:33 on dist-test-slave-kzfr"
server_key: "7b5cbd5de9e08a5d4b7e92e7e5fc87f7"
server_key_iv: "ebf38a0db866824011792ce97e3b877f"
server_key_version: "encryptionkey@0"
I20260928 22:57:33.066004 493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:33.088153 493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:33.088518 493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:33.088671 493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:33.092965 493 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:44621
I20260928 22:57:33.093017 559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:44621 every 8 connection(s)
I20260928 22:57:33.093385 493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:33.094455 560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:33.096937 560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1: Bootstrap starting.
I20260928 22:57:33.097493 560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:33.097815 560 log.cc:824] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:33.098446 560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1: No bootstrap required, opened a new log
I20260928 22:57:33.100219 560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962e1b7171f143ffa08fd12ddbba72b1" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44621 } }
I20260928 22:57:33.100457 560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:33.100493 560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962e1b7171f143ffa08fd12ddbba72b1, State: Initialized, Role: FOLLOWER
I20260928 22:57:33.100639 560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962e1b7171f143ffa08fd12ddbba72b1" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44621 } }
I20260928 22:57:33.100723 560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:33.100746 560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:33.100775 560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:33.101248 560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962e1b7171f143ffa08fd12ddbba72b1" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44621 } }
I20260928 22:57:33.101338 560 leader_election.cc:302] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 962e1b7171f143ffa08fd12ddbba72b1; no voters:
I20260928 22:57:33.101549 560 leader_election.cc:288] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:33.101636 563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:33.101812 563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [term 1 LEADER]: Becoming Leader. State: Replica: 962e1b7171f143ffa08fd12ddbba72b1, State: Running, Role: LEADER
I20260928 22:57:33.101859 560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:33.101912 563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962e1b7171f143ffa08fd12ddbba72b1" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44621 } }
I20260928 22:57:33.102246 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 493
I20260928 22:57:33.102360 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:33.102387 565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 962e1b7171f143ffa08fd12ddbba72b1. Latest consensus state: current_term: 1 leader_uuid: "962e1b7171f143ffa08fd12ddbba72b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962e1b7171f143ffa08fd12ddbba72b1" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44621 } } }
I20260928 22:57:33.102463 565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:33.102288 564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "962e1b7171f143ffa08fd12ddbba72b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962e1b7171f143ffa08fd12ddbba72b1" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44621 } } }
I20260928 22:57:33.102602 14394 external_mini_cluster.cc:1428] Setting key 51769777c3caa0776154b8cdcfd6addd
I20260928 22:57:33.102634 564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:33.102658 567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:33.103484 567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:33.105032 567 catalog_manager.cc:1347] Generated new cluster ID: 6bde33cba8e9456e9fe4c1b10a6db7ee
I20260928 22:57:33.105091 567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:33.108918 572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.103129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48872 (local address 127.14.14.190:44621)
0928 22:57:33.103354 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:33.103365 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:33.103417 (+ 52us) server_negotiation.cc:407] Connection header received
0928 22:57:33.103544 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:33.103552 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:33.103632 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:33.103837 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:33.104139 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.105672 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.106743 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.107368 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.107495 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.108194 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:33.108198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:33.108198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:33.108409 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:33.108452 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:33.108468 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:33.108536 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:33.108622 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 22:57:33.108700 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260928 22:57:33.117638 567 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:33.118315 567 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:33.124354 567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1: Generated new TSK 0
I20260928 22:57:33.124514 567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:33.132012 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44621
--builtin_ntp_servers=127.14.14.148:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:57:33.214685 584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:33.214896 584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:33.214918 584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:33.216073 584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:33.216121 584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:33.216147 584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:57:33.217514 584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:33.217926 584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:33.218381 584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:33.224690 590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:33.224694 592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:33.224767 584 server_base.cc:1034] running on GCE node
W20260928 22:57:33.224903 589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:33.225096 584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:33.225363 584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:33.226528 584 hybrid_clock.cc:648] HybridClock initialized: now 1790636253226519 us; error 28 us; skew 500 ppm
I20260928 22:57:33.227828 584 webserver.cc:460] Webserver started at http://127.14.14.129:36261/ using document root <none> and password file <none>
I20260928 22:57:33.228071 584 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:33.228120 584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:33.228276 584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:33.229351 584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "8b61ce679b914a62823ff6168cfa2d9e"
format_stamp: "Formatted at 2026-09-28 22:57:33 on dist-test-slave-kzfr"
server_key: "4396e99918f015ca0de9d41e788309f1"
server_key_iv: "ce4f7005d387af8972b26f12c883ca4c"
server_key_version: "encryptionkey@0"
I20260928 22:57:33.229737 584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "8b61ce679b914a62823ff6168cfa2d9e"
format_stamp: "Formatted at 2026-09-28 22:57:33 on dist-test-slave-kzfr"
server_key: "4396e99918f015ca0de9d41e788309f1"
server_key_iv: "ce4f7005d387af8972b26f12c883ca4c"
server_key_version: "encryptionkey@0"
I20260928 22:57:33.231087 584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:57:33.231875 598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:33.232031 584 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:57:33.232131 584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "8b61ce679b914a62823ff6168cfa2d9e"
format_stamp: "Formatted at 2026-09-28 22:57:33 on dist-test-slave-kzfr"
server_key: "4396e99918f015ca0de9d41e788309f1"
server_key_iv: "ce4f7005d387af8972b26f12c883ca4c"
server_key_version: "encryptionkey@0"
I20260928 22:57:33.232201 584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:33.250881 584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:33.251184 584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:33.251327 584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:33.251551 584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:33.251884 584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:33.251917 584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:33.251940 584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:33.251981 584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:33.257782 584 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:41739
I20260928 22:57:33.257858 711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:41739 every 8 connection(s)
I20260928 22:57:33.258172 584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:57:33.263187 572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.258921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:59933 (local address 127.14.14.190:44621)
0928 22:57:33.258985 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:33.258987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:33.259211 (+ 224us) server_negotiation.cc:407] Connection header received
0928 22:57:33.259701 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:33.259702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:33.259713 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:33.259758 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:33.260371 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.260862 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.261924 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.262112 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.262168 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.262991 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:33.262992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:33.262992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:33.262996 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:33.263006 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:33.263009 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:33.263045 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:33.263106 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 22:57:33.263129 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:57:33.263413 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.258885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44621 (local address 127.14.14.129:59933)
0928 22:57:33.259082 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:33.259123 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:33.259601 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:33.259811 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:33.259814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:33.259821 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:33.260302 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.260306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.260954 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.260954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.261854 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.261857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.262191 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.262192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.262301 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.262859 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:57:33.262859 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:33.262887 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:57:33.262922 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:33.263047 (+ 125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:33.263050 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:33.263052 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:57:33.263098 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 22:57:33.263170 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260928 22:57:33.263957 712 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44621
I20260928 22:57:33.264055 712 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:33.264235 712 heartbeater.cc:511] Master 127.14.14.190:44621 requested a full tablet report, sending...
I20260928 22:57:33.264715 524 ts_manager.cc:194] Registered new tserver with Master: 8b61ce679b914a62823ff6168cfa2d9e (127.14.14.129:41739)
I20260928 22:57:33.265323 524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.14.129:59933
I20260928 22:57:33.266927 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 584
I20260928 22:57:33.267007 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:57:33.267174 14394 external_mini_cluster.cc:1428] Setting key 69bcc3b332da3fe027c3fe3452a923db
I20260928 22:57:33.268024 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44621
--builtin_ntp_servers=127.14.14.148:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:57:33.349442 715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:33.349668 715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:33.349690 715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:33.350765 715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:33.350796 715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:33.350821 715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:57:33.352082 715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:33.352326 715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:33.352636 715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:33.357615 721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:33.357640 720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:33.357640 723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:33.357807 715 server_base.cc:1034] running on GCE node
I20260928 22:57:33.358033 715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:33.358238 715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:33.359481 715 hybrid_clock.cc:648] HybridClock initialized: now 1790636253359461 us; error 38 us; skew 500 ppm
I20260928 22:57:33.360790 715 webserver.cc:460] Webserver started at http://127.14.14.130:44535/ using document root <none> and password file <none>
I20260928 22:57:33.361050 715 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:33.361111 715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:33.361244 715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:33.362352 715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "4cb57456170b4cf4bb3ca24ae221c00d"
format_stamp: "Formatted at 2026-09-28 22:57:33 on dist-test-slave-kzfr"
server_key: "c012c4778146ba162d9eb74ff0f8c4f9"
server_key_iv: "a4ee821ae7c61e4090abe397d847eec1"
server_key_version: "encryptionkey@0"
I20260928 22:57:33.362699 715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "4cb57456170b4cf4bb3ca24ae221c00d"
format_stamp: "Formatted at 2026-09-28 22:57:33 on dist-test-slave-kzfr"
server_key: "c012c4778146ba162d9eb74ff0f8c4f9"
server_key_iv: "a4ee821ae7c61e4090abe397d847eec1"
server_key_version: "encryptionkey@0"
I20260928 22:57:33.364009 715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:57:33.364799 729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:33.365000 715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:57:33.365092 715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "4cb57456170b4cf4bb3ca24ae221c00d"
format_stamp: "Formatted at 2026-09-28 22:57:33 on dist-test-slave-kzfr"
server_key: "c012c4778146ba162d9eb74ff0f8c4f9"
server_key_iv: "a4ee821ae7c61e4090abe397d847eec1"
server_key_version: "encryptionkey@0"
I20260928 22:57:33.365152 715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:33.388082 715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:33.388379 715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:33.388494 715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:33.388727 715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:33.389046 715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:33.389078 715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:33.389118 715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:33.389137 715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:33.394989 715 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:34907
I20260928 22:57:33.395056 849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:34907 every 8 connection(s)
I20260928 22:57:33.395392 715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:57:33.400935 572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.396182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:56041 (local address 127.14.14.190:44621)
0928 22:57:33.396250 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:33.396252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:33.396533 (+ 281us) server_negotiation.cc:407] Connection header received
0928 22:57:33.397093 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:33.397095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:33.397107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:33.397175 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:33.397955 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.398470 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.399484 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.399681 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.399763 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.400637 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:33.400639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:33.400639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:33.400644 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:33.400659 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:33.400663 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:33.400726 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:33.400821 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 22:57:33.400852 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:57:33.401171 852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.396185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44621 (local address 127.14.14.130:56041)
0928 22:57:33.396381 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:33.396424 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:33.396980 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:33.397249 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:33.397252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:33.397260 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:33.397844 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.397849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.398570 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.398572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.399378 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.399380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.399774 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.399775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.399915 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.400484 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:57:33.400485 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:33.400515 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:57:33.400539 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:33.400727 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:33.400734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:33.400737 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:57:33.400823 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 22:57:33.400938 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260928 22:57:33.401695 850 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44621
I20260928 22:57:33.401808 850 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:33.402021 850 heartbeater.cc:511] Master 127.14.14.190:44621 requested a full tablet report, sending...
I20260928 22:57:33.402401 524 ts_manager.cc:194] Registered new tserver with Master: 4cb57456170b4cf4bb3ca24ae221c00d (127.14.14.130:34907)
I20260928 22:57:33.402798 524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.14.130:56041
I20260928 22:57:33.403026 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 715
I20260928 22:57:33.403143 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:57:33.403359 14394 external_mini_cluster.cc:1428] Setting key ea38ee5dab6c903c07b49d65dad2eed3
I20260928 22:57:33.404277 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44621
--builtin_ntp_servers=127.14.14.148:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:57:33.484836 853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:33.485061 853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:33.485092 853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:33.486212 853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:33.486251 853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:33.486275 853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:57:33.487550 853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:33.487753 853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:33.488001 853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:33.493270 858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:33.493284 861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:33.493279 859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:33.493709 853 server_base.cc:1034] running on GCE node
I20260928 22:57:33.493934 853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:33.494204 853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:33.495352 853 hybrid_clock.cc:648] HybridClock initialized: now 1790636253495339 us; error 32 us; skew 500 ppm
I20260928 22:57:33.496634 853 webserver.cc:460] Webserver started at http://127.14.14.131:34647/ using document root <none> and password file <none>
I20260928 22:57:33.496896 853 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:33.496941 853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:33.497112 853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:33.498281 853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "39f7ad6d3d104640a2c9d4978b830d59"
format_stamp: "Formatted at 2026-09-28 22:57:33 on dist-test-slave-kzfr"
server_key: "f1ff9c90d83fb04a604cdb5062fc9253"
server_key_iv: "9e3df5e0d89f8ffd58224cd2b0860fc7"
server_key_version: "encryptionkey@0"
I20260928 22:57:33.498591 853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "39f7ad6d3d104640a2c9d4978b830d59"
format_stamp: "Formatted at 2026-09-28 22:57:33 on dist-test-slave-kzfr"
server_key: "f1ff9c90d83fb04a604cdb5062fc9253"
server_key_iv: "9e3df5e0d89f8ffd58224cd2b0860fc7"
server_key_version: "encryptionkey@0"
I20260928 22:57:33.499922 853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:57:33.500712 867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:33.500895 853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:33.500993 853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "39f7ad6d3d104640a2c9d4978b830d59"
format_stamp: "Formatted at 2026-09-28 22:57:33 on dist-test-slave-kzfr"
server_key: "f1ff9c90d83fb04a604cdb5062fc9253"
server_key_iv: "9e3df5e0d89f8ffd58224cd2b0860fc7"
server_key_version: "encryptionkey@0"
I20260928 22:57:33.501070 853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:33.522163 853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:33.522490 853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:33.522641 853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:33.522876 853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:33.523221 853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:33.523273 853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:33.523316 853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:33.523345 853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:33.528913 853 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:33379
I20260928 22:57:33.528963 980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:33379 every 8 connection(s)
I20260928 22:57:33.529294 853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:57:33.534615 572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.530159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:52933 (local address 127.14.14.190:44621)
0928 22:57:33.530244 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:33.530247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:33.530436 (+ 189us) server_negotiation.cc:407] Connection header received
0928 22:57:33.530979 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:33.530980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:33.530991 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:33.531034 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:33.531667 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.532163 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.533208 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.533380 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.533430 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.534396 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:33.534398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:33.534398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:33.534402 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:33.534415 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:33.534420 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:33.534468 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:33.534533 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 22:57:33.534557 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 22:57:33.534818 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.530116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44621 (local address 127.14.14.131:52933)
0928 22:57:33.530315 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:33.530351 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:33.530880 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:33.531091 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:33.531094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:33.531103 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:33.531603 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.531607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.532265 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.532266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.533131 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.533134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.533507 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.533508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.533638 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.534259 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:57:33.534260 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:33.534288 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:57:33.534311 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:33.534478 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:33.534482 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:33.534484 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:57:33.534536 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 22:57:33.534614 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
I20260928 22:57:33.535317 981 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44621
I20260928 22:57:33.535436 981 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:33.535621 981 heartbeater.cc:511] Master 127.14.14.190:44621 requested a full tablet report, sending...
I20260928 22:57:33.535980 524 ts_manager.cc:194] Registered new tserver with Master: 39f7ad6d3d104640a2c9d4978b830d59 (127.14.14.131:33379)
I20260928 22:57:33.536301 524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.14.131:52933
I20260928 22:57:33.539433 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 853
I20260928 22:57:33.539515 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:57:33.539685 14394 external_mini_cluster.cc:1428] Setting key dbd5b6baf2159a604a66f17a48d6b879
I20260928 22:57:33.540567 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:57:33.544739 572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.542031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.14.14.190:44621)
0928 22:57:33.542089 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:33.542091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:33.542186 (+ 95us) server_negotiation.cc:407] Connection header received
0928 22:57:33.542234 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:33.542235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:33.542245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:33.542292 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:33.542436 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.542906 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.543651 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.543820 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.543867 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.544560 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:33.544561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:33.544561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:33.544565 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:33.544574 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:33.544577 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:33.544636 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:33.544678 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 22:57:33.544687 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:57:33.546334 524 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:57:33.547809 524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:57:33.553975 994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.550370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.14.14.131:33379)
0928 22:57:33.550526 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:33.550531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:33.550540 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:57:33.550573 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:33.550574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:33.550588 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:33.550669 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:33.550826 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.551576 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.552792 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.553537 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.553659 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.553758 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:33.553845 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 22:57:33.553884 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260928 22:57:33.553962 993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.550171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:33379 (local address 127.0.0.1:53934)
0928 22:57:33.550382 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:33.550392 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:33.550433 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:33.550684 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:33.550687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:33.550689 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:33.550754 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.550756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.551669 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.551670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.552693 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.552695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.553661 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.553662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.553739 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.553752 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 22:57:33.553831 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 22:57:33.553885 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":63,"threads_started":1}
I20260928 22:57:33.554421 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.549936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:34907 (local address 127.0.0.1:42112)
0928 22:57:33.550226 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:33.550269 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:33.550347 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:33.550838 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:33.550839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:33.550841 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:33.550874 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.550876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.552191 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.552192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.553109 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.553112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.554191 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.554193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.554256 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.554260 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:57:33.554316 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:57:33.554356 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":71,"threads_started":1}
I20260928 22:57:33.554760 915 tablet_service.cc:1467] Processing CreateTablet for tablet d96eaa5ba6f84f7cbc630b5836ca8278 (DEFAULT_TABLE table=test-table [id=3220277103254cdcbeb47c8355622b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:33.554994 989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.549940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:41739 (local address 127.0.0.1:34668)
0928 22:57:33.550239 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:33.550269 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:33.550348 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:33.550891 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:33.550892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:33.550895 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:33.550938 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.550940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.552256 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.552257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.553304 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.553307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.554766 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.554766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.554814 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.554816 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:57:33.554871 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:57:33.554912 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":98,"threads_started":1}
I20260928 22:57:33.555186 915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96eaa5ba6f84f7cbc630b5836ca8278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:33.555218 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.550049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.14.14.129:41739)
0928 22:57:33.550282 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:33.550289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:33.550732 (+ 443us) server_negotiation.cc:407] Connection header received
0928 22:57:33.550774 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:33.550777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:33.550792 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:33.551054 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:33.551120 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.552166 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.553392 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.554689 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.554766 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.554852 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:33.554881 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 22:57:33.554918 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":99,"threads_started":1}
I20260928 22:57:33.555801 646 tablet_service.cc:1467] Processing CreateTablet for tablet d96eaa5ba6f84f7cbc630b5836ca8278 (DEFAULT_TABLE table=test-table [id=3220277103254cdcbeb47c8355622b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:33.556110 646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96eaa5ba6f84f7cbc630b5836ca8278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:33.557958 995 tablet_bootstrap.cc:492] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59: Bootstrap starting.
I20260928 22:57:33.558159 991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.550038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42112 (local address 127.14.14.130:34907)
0928 22:57:33.550207 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:33.550212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:33.550517 (+ 305us) server_negotiation.cc:407] Connection header received
0928 22:57:33.550567 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:33.550569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:33.550586 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:33.550670 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:33.550971 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.552090 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.553193 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.554098 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.554186 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.554266 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:33.558023 (+ 3757us) server_negotiation.cc:299] Negotiation successful
0928 22:57:33.558076 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260928 22:57:33.558593 995 tablet_bootstrap.cc:654] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:33.558919 995 log.cc:824] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:33.558951 996 tablet_bootstrap.cc:492] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e: Bootstrap starting.
I20260928 22:57:33.559561 996 tablet_bootstrap.cc:654] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:33.559531 784 tablet_service.cc:1467] Processing CreateTablet for tablet d96eaa5ba6f84f7cbc630b5836ca8278 (DEFAULT_TABLE table=test-table [id=3220277103254cdcbeb47c8355622b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:33.559743 995 tablet_bootstrap.cc:492] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59: No bootstrap required, opened a new log
I20260928 22:57:33.559813 995 ts_tablet_manager.cc:1397] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:57:33.559881 784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96eaa5ba6f84f7cbc630b5836ca8278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:33.559877 996 log.cc:824] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:33.560530 996 tablet_bootstrap.cc:492] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e: No bootstrap required, opened a new log
I20260928 22:57:33.560609 996 ts_tablet_manager.cc:1397] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:57:33.562474 996 raft_consensus.cc:348] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb57456170b4cf4bb3ca24ae221c00d" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34907 } } peers { permanent_uuid: "8b61ce679b914a62823ff6168cfa2d9e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41739 } } peers { permanent_uuid: "39f7ad6d3d104640a2c9d4978b830d59" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33379 } }
I20260928 22:57:33.562605 996 raft_consensus.cc:374] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:33.562572 995 raft_consensus.cc:348] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb57456170b4cf4bb3ca24ae221c00d" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34907 } } peers { permanent_uuid: "8b61ce679b914a62823ff6168cfa2d9e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41739 } } peers { permanent_uuid: "39f7ad6d3d104640a2c9d4978b830d59" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33379 } }
I20260928 22:57:33.562738 995 raft_consensus.cc:374] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:33.562791 995 raft_consensus.cc:729] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f7ad6d3d104640a2c9d4978b830d59, State: Initialized, Role: FOLLOWER
I20260928 22:57:33.562674 996 raft_consensus.cc:729] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b61ce679b914a62823ff6168cfa2d9e, State: Initialized, Role: FOLLOWER
I20260928 22:57:33.562935 995 consensus_queue.cc:260] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb57456170b4cf4bb3ca24ae221c00d" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34907 } } peers { permanent_uuid: "8b61ce679b914a62823ff6168cfa2d9e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41739 } } peers { permanent_uuid: "39f7ad6d3d104640a2c9d4978b830d59" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33379 } }
I20260928 22:57:33.562999 996 consensus_queue.cc:260] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb57456170b4cf4bb3ca24ae221c00d" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34907 } } peers { permanent_uuid: "8b61ce679b914a62823ff6168cfa2d9e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41739 } } peers { permanent_uuid: "39f7ad6d3d104640a2c9d4978b830d59" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33379 } }
I20260928 22:57:33.563248 995 ts_tablet_manager.cc:1428] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 22:57:33.563272 996 ts_tablet_manager.cc:1428] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:57:33.563294 712 heartbeater.cc:503] Master 127.14.14.190:44621 was elected leader, sending a full tablet report...
I20260928 22:57:33.563577 981 heartbeater.cc:503] Master 127.14.14.190:44621 was elected leader, sending a full tablet report...
I20260928 22:57:33.564007 1001 tablet_bootstrap.cc:492] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d: Bootstrap starting.
I20260928 22:57:33.564863 1001 tablet_bootstrap.cc:654] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:33.565143 1001 log.cc:824] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:33.565881 1001 tablet_bootstrap.cc:492] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d: No bootstrap required, opened a new log
I20260928 22:57:33.565979 1001 ts_tablet_manager.cc:1397] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:57:33.567906 1001 raft_consensus.cc:348] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb57456170b4cf4bb3ca24ae221c00d" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34907 } } peers { permanent_uuid: "8b61ce679b914a62823ff6168cfa2d9e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41739 } } peers { permanent_uuid: "39f7ad6d3d104640a2c9d4978b830d59" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33379 } }
I20260928 22:57:33.568060 1001 raft_consensus.cc:374] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:33.568104 1001 raft_consensus.cc:729] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb57456170b4cf4bb3ca24ae221c00d, State: Initialized, Role: FOLLOWER
I20260928 22:57:33.568221 1001 consensus_queue.cc:260] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb57456170b4cf4bb3ca24ae221c00d" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34907 } } peers { permanent_uuid: "8b61ce679b914a62823ff6168cfa2d9e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41739 } } peers { permanent_uuid: "39f7ad6d3d104640a2c9d4978b830d59" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33379 } }
I20260928 22:57:33.568501 1001 ts_tablet_manager.cc:1428] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:57:33.568543 850 heartbeater.cc:503] Master 127.14.14.190:44621 was elected leader, sending a full tablet report...
I20260928 22:57:33.575343 1000 raft_consensus.cc:482] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:57:33.575448 1000 raft_consensus.cc:504] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb57456170b4cf4bb3ca24ae221c00d" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34907 } } peers { permanent_uuid: "8b61ce679b914a62823ff6168cfa2d9e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41739 } } peers { permanent_uuid: "39f7ad6d3d104640a2c9d4978b830d59" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33379 } }
I20260928 22:57:33.575731 1000 leader_election.cc:288] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cb57456170b4cf4bb3ca24ae221c00d (127.14.14.130:34907), 39f7ad6d3d104640a2c9d4978b830d59 (127.14.14.131:33379)
I20260928 22:57:33.578734 991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.575986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:35687 (local address 127.14.14.130:34907)
0928 22:57:33.576037 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:33.576039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:33.576302 (+ 263us) server_negotiation.cc:407] Connection header received
0928 22:57:33.576330 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:33.576331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:33.576341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:33.576401 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:33.576566 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.577010 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.578013 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.578367 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.578421 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.578428 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:33.578588 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 22:57:33.578633 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:57:33.578738 994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.575977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:34471 (local address 127.14.14.131:33379)
0928 22:57:33.576025 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:33.576028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:33.576039 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:57:33.576098 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:33.576099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:33.576110 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:33.576181 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:33.576470 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.576984 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.578017 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.578381 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.578465 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.578473 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:33.578633 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 22:57:33.578674 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:57:33.578747 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.575923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:34907 (local address 127.14.14.129:35687)
0928 22:57:33.576121 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:33.576131 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:33.576189 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:33.576409 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:33.576410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:33.576411 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:33.576496 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.576497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.577116 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.577117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.577924 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.577926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.578443 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.578444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.578523 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.578526 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:57:33.578568 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 22:57:33.578625 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":16,"thread_start_us":93,"threads_started":1}
I20260928 22:57:33.578749 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.575911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:33379 (local address 127.14.14.129:34471)
0928 22:57:33.575971 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:33.575985 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:33.576030 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:33.576317 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:33.576317 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:33.576320 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:33.576394 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.576397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.577071 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.577072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.577921 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:33.577923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.578487 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.578488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:33.578546 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.578548 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:57:33.578619 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 22:57:33.578657 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":3}
I20260928 22:57:33.579160 804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96eaa5ba6f84f7cbc630b5836ca8278" candidate_uuid: "8b61ce679b914a62823ff6168cfa2d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cb57456170b4cf4bb3ca24ae221c00d" is_pre_election: true
I20260928 22:57:33.579231 935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96eaa5ba6f84f7cbc630b5836ca8278" candidate_uuid: "8b61ce679b914a62823ff6168cfa2d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f7ad6d3d104640a2c9d4978b830d59" is_pre_election: true
I20260928 22:57:33.579329 804 raft_consensus.cc:2454] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b61ce679b914a62823ff6168cfa2d9e in term 0.
I20260928 22:57:33.579406 935 raft_consensus.cc:2454] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b61ce679b914a62823ff6168cfa2d9e in term 0.
I20260928 22:57:33.579576 600 leader_election.cc:302] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cb57456170b4cf4bb3ca24ae221c00d, 8b61ce679b914a62823ff6168cfa2d9e; no voters:
I20260928 22:57:33.579756 1000 raft_consensus.cc:2781] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:57:33.579818 1000 raft_consensus.cc:482] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:57:33.579855 1000 raft_consensus.cc:3037] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:33.580581 1000 raft_consensus.cc:504] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb57456170b4cf4bb3ca24ae221c00d" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34907 } } peers { permanent_uuid: "8b61ce679b914a62823ff6168cfa2d9e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41739 } } peers { permanent_uuid: "39f7ad6d3d104640a2c9d4978b830d59" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33379 } }
I20260928 22:57:33.580714 1000 leader_election.cc:288] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [CANDIDATE]: Term 1 election: Requested vote from peers 4cb57456170b4cf4bb3ca24ae221c00d (127.14.14.130:34907), 39f7ad6d3d104640a2c9d4978b830d59 (127.14.14.131:33379)
I20260928 22:57:33.580897 935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96eaa5ba6f84f7cbc630b5836ca8278" candidate_uuid: "8b61ce679b914a62823ff6168cfa2d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f7ad6d3d104640a2c9d4978b830d59"
I20260928 22:57:33.580967 935 raft_consensus.cc:3037] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:33.581839 935 raft_consensus.cc:2454] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b61ce679b914a62823ff6168cfa2d9e in term 1.
I20260928 22:57:33.581954 804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96eaa5ba6f84f7cbc630b5836ca8278" candidate_uuid: "8b61ce679b914a62823ff6168cfa2d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cb57456170b4cf4bb3ca24ae221c00d"
I20260928 22:57:33.582005 602 leader_election.cc:302] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f7ad6d3d104640a2c9d4978b830d59, 8b61ce679b914a62823ff6168cfa2d9e; no voters:
I20260928 22:57:33.582029 804 raft_consensus.cc:3037] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:33.582124 1000 raft_consensus.cc:2781] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:33.582281 1000 raft_consensus.cc:686] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [term 1 LEADER]: Becoming Leader. State: Replica: 8b61ce679b914a62823ff6168cfa2d9e, State: Running, Role: LEADER
I20260928 22:57:33.582377 1000 consensus_queue.cc:237] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb57456170b4cf4bb3ca24ae221c00d" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34907 } } peers { permanent_uuid: "8b61ce679b914a62823ff6168cfa2d9e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41739 } } peers { permanent_uuid: "39f7ad6d3d104640a2c9d4978b830d59" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33379 } }
I20260928 22:57:33.582980 804 raft_consensus.cc:2454] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b61ce679b914a62823ff6168cfa2d9e in term 1.
I20260928 22:57:33.583264 523 catalog_manager.cc:5495] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b61ce679b914a62823ff6168cfa2d9e (127.14.14.129). New cstate: current_term: 1 leader_uuid: "8b61ce679b914a62823ff6168cfa2d9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb57456170b4cf4bb3ca24ae221c00d" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b61ce679b914a62823ff6168cfa2d9e" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 41739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f7ad6d3d104640a2c9d4978b830d59" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33379 } health_report { overall_health: UNKNOWN } } }
I20260928 22:57:33.591748 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.589349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34684 (local address 127.14.14.129:41739)
0928 22:57:33.589402 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:33.589405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:33.589413 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:57:33.589477 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:33.589478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:33.589496 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:33.589560 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:57:33.589745 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.590215 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.590967 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:33.591182 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:33.591254 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:33.591401 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:57:33.591495 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:57:33.591608 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:33.591638 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 22:57:33.591648 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:57:33.593465 804 raft_consensus.cc:1261] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d [term 1 FOLLOWER]: Refusing update from remote peer 8b61ce679b914a62823ff6168cfa2d9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:57:33.593466 935 raft_consensus.cc:1261] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59 [term 1 FOLLOWER]: Refusing update from remote peer 8b61ce679b914a62823ff6168cfa2d9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:57:33.593870 1000 consensus_queue.cc:1035] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f7ad6d3d104640a2c9d4978b830d59" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 33379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:57:33.594002 1005 consensus_queue.cc:1035] T d96eaa5ba6f84f7cbc630b5836ca8278 P 8b61ce679b914a62823ff6168cfa2d9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cb57456170b4cf4bb3ca24ae221c00d" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:57:33.596576 1008 mvcc.cc:205] Tried to move back new op lower bound from 7334446094716338176 to 7334446094674284544. Current Snapshot: MvccSnapshot[applied={T|T < 7334446094716338176}]
I20260928 22:57:33.598966 523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:57:33.599054 523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:57:33.599820 523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 962e1b7171f143ffa08fd12ddbba72b1: Sending DeleteTablet for 3 replicas of tablet d96eaa5ba6f84f7cbc630b5836ca8278
I20260928 22:57:33.600147 646 tablet_service.cc:1514] Processing DeleteTablet for tablet d96eaa5ba6f84f7cbc630b5836ca8278 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:33 UTC) from {username='slave'} at 127.0.0.1:34668
I20260928 22:57:33.600190 915 tablet_service.cc:1514] Processing DeleteTablet for tablet d96eaa5ba6f84f7cbc630b5836ca8278 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:33 UTC) from {username='slave'} at 127.0.0.1:53934
I20260928 22:57:33.600219 784 tablet_service.cc:1514] Processing DeleteTablet for tablet d96eaa5ba6f84f7cbc630b5836ca8278 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:33 UTC) from {username='slave'} at 127.0.0.1:42112
I20260928 22:57:33.600417 1019 tablet_replica.cc:330] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59: stopping tablet replica
I20260928 22:57:33.600517 1019 raft_consensus.cc:2229] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:57:33.600620 1019 raft_consensus.cc:2258] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:57:33.600622 1020 tablet_replica.cc:330] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d: stopping tablet replica
I20260928 22:57:33.600716 1020 raft_consensus.cc:2229] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:57:33.600870 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 584
I20260928 22:57:33.601997 1019 ts_tablet_manager.cc:1905] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:57:33.600805 1020 raft_consensus.cc:2258] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:57:33.603832 1019 ts_tablet_manager.cc:1918] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:57:33.603878 1020 ts_tablet_manager.cc:1905] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:57:33.604024 1019 log.cc:1196] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/d96eaa5ba6f84f7cbc630b5836ca8278
I20260928 22:57:33.604385 1019 ts_tablet_manager.cc:1939] T d96eaa5ba6f84f7cbc630b5836ca8278 P 39f7ad6d3d104640a2c9d4978b830d59: Deleting consensus metadata
I20260928 22:57:33.604872 511 catalog_manager.cc:4841] TS 39f7ad6d3d104640a2c9d4978b830d59 (127.14.14.131:33379): tablet d96eaa5ba6f84f7cbc630b5836ca8278 (table test-table [id=3220277103254cdcbeb47c8355622b27]) successfully deleted
I20260928 22:57:33.605481 1020 ts_tablet_manager.cc:1918] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:57:33.605543 1020 log.cc:1196] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/d96eaa5ba6f84f7cbc630b5836ca8278
I20260928 22:57:33.605780 1020 ts_tablet_manager.cc:1939] T d96eaa5ba6f84f7cbc630b5836ca8278 P 4cb57456170b4cf4bb3ca24ae221c00d: Deleting consensus metadata
I20260928 22:57:33.606127 509 catalog_manager.cc:4841] TS 4cb57456170b4cf4bb3ca24ae221c00d (127.14.14.130:34907): tablet d96eaa5ba6f84f7cbc630b5836ca8278 (table test-table [id=3220277103254cdcbeb47c8355622b27]) successfully deleted
W20260928 22:57:33.607352 511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:41739 (error 108)
I20260928 22:57:33.607622 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 715
I20260928 22:57:33.608405 989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:33.607765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:41739 (local address 127.0.0.1:34696)
0928 22:57:33.608322 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:33.608366 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536}
W20260928 22:57:33.608610 511 catalog_manager.cc:4584] TS 8b61ce679b914a62823ff6168cfa2d9e (127.14.14.129:41739): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d96eaa5ba6f84f7cbc630b5836ca8278: Network error: Client connection negotiation failed: client connection to 127.14.14.129:41739: connect: Connection refused (error 111)
I20260928 22:57:33.612440 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 853
I20260928 22:57:33.617249 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 493
2026-09-28T22:57:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (675 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:33 dist-test-slave-kzfr krb5kdc[1024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:33 dist-test-slave-kzfr krb5kdc[1024](info): set up 2 sockets
Sep 28 22:57:33 dist-test-slave-kzfr krb5kdc[1024](info): commencing operation
krb5kdc: starting...
W20260928 22:57:34.653752 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:34 dist-test-slave-kzfr krb5kdc[1024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:57:34.772815 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44163
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:44163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:34.853371 1038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:34.853648 1038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:34.853689 1038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:34.854880 1038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:34.854946 1038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:34.854969 1038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:34.854991 1038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:34.855020 1038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:34.856482 1038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:44163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:34.856766 1038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:34.856989 1038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:34.861932 1044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:34.861956 1046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:34.861950 1043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:34.861985 1038 server_base.cc:1034] running on GCE node
I20260928 22:57:34.862445 1038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:34.862757 1038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:34.863942 1038 hybrid_clock.cc:648] HybridClock initialized: now 1790636254863921 us; error 32 us; skew 500 ppm
Sep 28 22:57:34 dist-test-slave-kzfr krb5kdc[1024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636254, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:34.866688 1038 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:57:34.867175 1038 webserver.cc:460] Webserver started at http://127.14.14.190:46763/ using document root <none> and password file <none>
I20260928 22:57:34.867396 1038 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:34.867461 1038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:34.867604 1038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:34.868659 1038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "84fd0731a2a94becbd897b78ba9f3103"
format_stamp: "Formatted at 2026-09-28 22:57:34 on dist-test-slave-kzfr"
server_key: "987a9ef4c975615490fdcd62f2fe0d13"
server_key_iv: "7c4a5a2e4fe917d925c30fe6352ebb15"
server_key_version: "encryptionkey@0"
I20260928 22:57:34.869012 1038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "84fd0731a2a94becbd897b78ba9f3103"
format_stamp: "Formatted at 2026-09-28 22:57:34 on dist-test-slave-kzfr"
server_key: "987a9ef4c975615490fdcd62f2fe0d13"
server_key_iv: "7c4a5a2e4fe917d925c30fe6352ebb15"
server_key_version: "encryptionkey@0"
I20260928 22:57:34.870476 1038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:57:34.871184 1053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:34.871362 1038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:57:34.871464 1038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "84fd0731a2a94becbd897b78ba9f3103"
format_stamp: "Formatted at 2026-09-28 22:57:34 on dist-test-slave-kzfr"
server_key: "987a9ef4c975615490fdcd62f2fe0d13"
server_key_iv: "7c4a5a2e4fe917d925c30fe6352ebb15"
server_key_version: "encryptionkey@0"
I20260928 22:57:34.871533 1038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:34.880810 1038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:34.882709 1038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:34.882867 1038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:34.886749 1038 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:44163
I20260928 22:57:34.886787 1110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:44163 every 8 connection(s)
I20260928 22:57:34.887141 1038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:34.887466 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 1038
I20260928 22:57:34.887617 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:34.887851 14394 external_mini_cluster.cc:1428] Setting key b250b4dee35f4b7ebad7e748d8d42739
I20260928 22:57:34.888264 1111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:34.891907 1111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103: Bootstrap starting.
I20260928 22:57:34.892649 1111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:34.892975 1111 log.cc:824] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:34.893849 1111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103: No bootstrap required, opened a new log
Sep 28 22:57:34 dist-test-slave-kzfr krb5kdc[1024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:57:34.895963 1111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fd0731a2a94becbd897b78ba9f3103" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44163 } }
I20260928 22:57:34.896106 1111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:34.896130 1111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84fd0731a2a94becbd897b78ba9f3103, State: Initialized, Role: FOLLOWER
I20260928 22:57:34.896275 1111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fd0731a2a94becbd897b78ba9f3103" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44163 } }
I20260928 22:57:34.896348 1111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:34.896371 1111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:34.896445 1111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:34.897147 1111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fd0731a2a94becbd897b78ba9f3103" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44163 } }
I20260928 22:57:34.897240 1111 leader_election.cc:302] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84fd0731a2a94becbd897b78ba9f3103; no voters:
I20260928 22:57:34.897501 1111 leader_election.cc:288] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:34.897529 1116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:34.897668 1116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [term 1 LEADER]: Becoming Leader. State: Replica: 84fd0731a2a94becbd897b78ba9f3103, State: Running, Role: LEADER
I20260928 22:57:34.897755 1116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fd0731a2a94becbd897b78ba9f3103" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44163 } }
I20260928 22:57:34.897821 1111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:34.898079 1116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84fd0731a2a94becbd897b78ba9f3103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fd0731a2a94becbd897b78ba9f3103" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44163 } } }
I20260928 22:57:34.898149 1116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:34.898303 1116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84fd0731a2a94becbd897b78ba9f3103. Latest consensus state: current_term: 1 leader_uuid: "84fd0731a2a94becbd897b78ba9f3103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fd0731a2a94becbd897b78ba9f3103" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44163 } } }
I20260928 22:57:34.898370 1116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:34.898720 1121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:34.899231 1121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:34.899796 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:34.888393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.14.14.190:44163)
0928 22:57:34.888567 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:34.888577 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:34.888608 (+ 31us) server_negotiation.cc:407] Connection header received
0928 22:57:34.889027 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:34.889033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:34.889082 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:34.889195 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:34.890097 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:34.890977 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:34.892083 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:34.892494 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:34.892573 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:34.895494 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:34.895499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:34.895499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:34.895511 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:34.897695 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:34.898142 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:34.898145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:34.898145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:34.898288 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:34.898538 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:34.898539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:34.898540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:34.898893 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:34.899081 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:34.899469 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 22:57:34.899607 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260928 22:57:34.900890 1121 catalog_manager.cc:1347] Generated new cluster ID: 7ad94a6611ca431e8eae0cca6f8c06e8
I20260928 22:57:34.900941 1121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:34.907300 1121 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:34.907796 1121 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:34.912611 1121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103: Generated new TSK 0
I20260928 22:57:34.912796 1121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.14.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
Entry for principal kudu/127.14.14.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab.
I20260928 22:57:34.941105 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44163
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:35.022902 1137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:35.023100 1137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:35.023123 1137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:35.024382 1137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:35.024425 1137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:35.024451 1137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:57:35.025988 1137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.129.keytab
--principal=kudu/127.14.14.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:35.026217 1137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:35.026436 1137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:35.032552 1143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:35.032612 1142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:35.032748 1137 server_base.cc:1034] running on GCE node
W20260928 22:57:35.032570 1145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:35.033056 1137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:35.033345 1137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:35.034538 1137 hybrid_clock.cc:648] HybridClock initialized: now 1790636255034516 us; error 47 us; skew 500 ppm
Sep 28 22:57:35 dist-test-slave-kzfr krb5kdc[1024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636255, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:35.036988 1137 init.cc:377] Logged in from keytab as kudu/127.14.14.129@KRBTEST.COM (short username kudu)
I20260928 22:57:35.037452 1137 webserver.cc:460] Webserver started at http://127.14.14.129:42257/ using document root <none> and password file <none>
I20260928 22:57:35.037721 1137 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:35.037788 1137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:35.037938 1137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:35.038976 1137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "494a6638978a4cb6ad42048d391de894"
format_stamp: "Formatted at 2026-09-28 22:57:35 on dist-test-slave-kzfr"
server_key: "9788257fbc0735086876d730754f4c08"
server_key_iv: "b1002fe2dce3d4ea8635426fed7fd97b"
server_key_version: "encryptionkey@0"
I20260928 22:57:35.039340 1137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "494a6638978a4cb6ad42048d391de894"
format_stamp: "Formatted at 2026-09-28 22:57:35 on dist-test-slave-kzfr"
server_key: "9788257fbc0735086876d730754f4c08"
server_key_iv: "b1002fe2dce3d4ea8635426fed7fd97b"
server_key_version: "encryptionkey@0"
I20260928 22:57:35.040711 1137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:57:35.041380 1152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:35.041569 1137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:35.041693 1137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "494a6638978a4cb6ad42048d391de894"
format_stamp: "Formatted at 2026-09-28 22:57:35 on dist-test-slave-kzfr"
server_key: "9788257fbc0735086876d730754f4c08"
server_key_iv: "b1002fe2dce3d4ea8635426fed7fd97b"
server_key_version: "encryptionkey@0"
I20260928 22:57:35.041774 1137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:35.052346 1137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:35.054690 1137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:35.054860 1137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:35.055111 1137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:35.055423 1137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:35.055473 1137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:35.055526 1137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:35.055549 1137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:35.061322 1137 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:45201
I20260928 22:57:35.061408 1265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:45201 every 8 connection(s)
I20260928 22:57:35.061793 1137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:57:35.065899 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 1137
I20260928 22:57:35.066056 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:57:35.066267 14394 external_mini_cluster.cc:1428] Setting key bda20f55962d1f22425cfd1a5f656622
Sep 28 22:57:35 dist-test-slave-kzfr krb5kdc[1024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636255, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.129@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:57:35.072145 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.062574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:47387 (local address 127.14.14.190:44163)
0928 22:57:35.062637 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:35.062639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:35.062840 (+ 201us) server_negotiation.cc:407] Connection header received
0928 22:57:35.063334 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:35.063335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:35.063344 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:35.063392 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:35.064345 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.064920 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.066024 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.066358 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.066452 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.068852 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:35.068854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:35.068855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:35.068863 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:35.070681 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:35.071101 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:35.071102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:35.071102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:35.071170 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:35.071404 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:35.071405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:35.071406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:35.071572 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:35.071660 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:35.071897 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 22:57:35.072002 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:57:35.072228 1268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.062554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44163 (local address 127.14.14.129:47387)
0928 22:57:35.062722 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:35.062759 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:35.063253 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:35.063454 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:35.063457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:35.064032 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:35.064237 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.064242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.065007 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.065007 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.065873 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.065875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.066470 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.066471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.066634 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.067198 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:57:35.067207 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:35.068756 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:35.070778 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:35.070784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:35.070786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:35.071024 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:35.071241 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:35.071242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:35.071242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:35.071339 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:35.071644 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:35.071649 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:35.071765 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 22:57:35.071865 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 22:57:35.072034 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260928 22:57:35.072723 1266 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44163
I20260928 22:57:35.072835 1266 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:35.073043 1266 heartbeater.cc:511] Master 127.14.14.190:44163 requested a full tablet report, sending...
I20260928 22:57:35.073652 1075 ts_manager.cc:194] Registered new tserver with Master: 494a6638978a4cb6ad42048d391de894 (127.14.14.129:45201)
I20260928 22:57:35.074579 1075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.129@KRBTEST.COM'} at 127.14.14.129:47387
WARNING: no policy specified for kudu/127.14.14.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
Entry for principal kudu/127.14.14.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab.
I20260928 22:57:35.092912 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44163
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:35.173399 1271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:35.173612 1271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:35.173635 1271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:35.174834 1271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:35.174875 1271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:35.174902 1271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:57:35.176370 1271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.130.keytab
--principal=kudu/127.14.14.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:35.176560 1271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:35.176728 1271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:35.181800 1276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:35.181807 1279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:35.181867 1271 server_base.cc:1034] running on GCE node
W20260928 22:57:35.181813 1277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:35.182228 1271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:35.182464 1271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:35.183601 1271 hybrid_clock.cc:648] HybridClock initialized: now 1790636255183586 us; error 36 us; skew 500 ppm
Sep 28 22:57:35 dist-test-slave-kzfr krb5kdc[1024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636255, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:35.186234 1271 init.cc:377] Logged in from keytab as kudu/127.14.14.130@KRBTEST.COM (short username kudu)
I20260928 22:57:35.186612 1271 webserver.cc:460] Webserver started at http://127.14.14.130:38559/ using document root <none> and password file <none>
I20260928 22:57:35.186787 1271 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:35.186826 1271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:35.186913 1271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:35.188011 1271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "9a0185d8746741329503c060ad30fda0"
format_stamp: "Formatted at 2026-09-28 22:57:35 on dist-test-slave-kzfr"
server_key: "9d9b7d9b4f03d096875ed98399f0c3b2"
server_key_iv: "1b26117e17af6dadb2dd48a7c9a2c9f4"
server_key_version: "encryptionkey@0"
I20260928 22:57:35.188306 1271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "9a0185d8746741329503c060ad30fda0"
format_stamp: "Formatted at 2026-09-28 22:57:35 on dist-test-slave-kzfr"
server_key: "9d9b7d9b4f03d096875ed98399f0c3b2"
server_key_iv: "1b26117e17af6dadb2dd48a7c9a2c9f4"
server_key_version: "encryptionkey@0"
I20260928 22:57:35.189661 1271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:57:35.190433 1286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:35.190659 1271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:35.190730 1271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "9a0185d8746741329503c060ad30fda0"
format_stamp: "Formatted at 2026-09-28 22:57:35 on dist-test-slave-kzfr"
server_key: "9d9b7d9b4f03d096875ed98399f0c3b2"
server_key_iv: "1b26117e17af6dadb2dd48a7c9a2c9f4"
server_key_version: "encryptionkey@0"
I20260928 22:57:35.190822 1271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:35.220443 1271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:35.222483 1271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:35.222642 1271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:35.222890 1271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:35.223210 1271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:35.223244 1271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:35.223268 1271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:35.223311 1271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:35.229300 1271 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:39425
I20260928 22:57:35.229379 1399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:39425 every 8 connection(s)
I20260928 22:57:35.229720 1271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:57:35 dist-test-slave-kzfr krb5kdc[1024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636255, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.130@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:57:35.238070 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 1271
I20260928 22:57:35.238202 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:57:35.238494 14394 external_mini_cluster.cc:1428] Setting key b7b157b16529fabcad74f3a9b3dae998
I20260928 22:57:35.239117 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.230445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:52735 (local address 127.14.14.190:44163)
0928 22:57:35.230526 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:35.230528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:35.230745 (+ 217us) server_negotiation.cc:407] Connection header received
0928 22:57:35.231269 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:35.231270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:35.231282 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:35.231339 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:35.232288 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.232787 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.233786 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.234000 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.234073 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.236159 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:35.236161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:35.236161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:35.236169 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:35.237570 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:35.238009 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:35.238010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:35.238010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:35.238068 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:35.238355 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:35.238356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:35.238357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:35.238511 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:35.238592 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:35.238927 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 22:57:35.239013 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:57:35.239255 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.230411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44163 (local address 127.14.14.130:52735)
0928 22:57:35.230625 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:35.230660 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:35.231164 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:35.231408 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:35.231412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:35.231980 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:35.232178 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.232183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.232878 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.232879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.233697 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.233699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.234077 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.234078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.234203 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.234716 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:57:35.234724 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:35.236063 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:35.237684 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:35.237691 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:35.237693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:35.237933 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:35.238140 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:35.238140 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:35.238141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:35.238261 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:35.238600 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:35.238606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:35.238741 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 22:57:35.238863 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 22:57:35.239008 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":95,"threads_started":1}
I20260928 22:57:35.239713 1400 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44163
I20260928 22:57:35.239826 1400 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:35.240023 1400 heartbeater.cc:511] Master 127.14.14.190:44163 requested a full tablet report, sending...
I20260928 22:57:35.240499 1075 ts_manager.cc:194] Registered new tserver with Master: 9a0185d8746741329503c060ad30fda0 (127.14.14.130:39425)
I20260928 22:57:35.240986 1075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.130@KRBTEST.COM'} at 127.14.14.130:52735
WARNING: no policy specified for kudu/127.14.14.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
Entry for principal kudu/127.14.14.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab.
I20260928 22:57:35.264565 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44163
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:35.346449 1405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:35.346689 1405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:35.346728 1405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:35.347851 1405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:35.347913 1405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:35.347967 1405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:57:35.349414 1405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.131.keytab
--principal=kudu/127.14.14.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:44163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:35.349694 1405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:35.349917 1405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:35.354775 1410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:35.354775 1413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:35.354805 1405 server_base.cc:1034] running on GCE node
W20260928 22:57:35.354797 1411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:35.355193 1405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:35.355386 1405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:35.356559 1405 hybrid_clock.cc:648] HybridClock initialized: now 1790636255356545 us; error 31 us; skew 500 ppm
Sep 28 22:57:35 dist-test-slave-kzfr krb5kdc[1024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636255, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:35.359267 1405 init.cc:377] Logged in from keytab as kudu/127.14.14.131@KRBTEST.COM (short username kudu)
I20260928 22:57:35.359750 1405 webserver.cc:460] Webserver started at http://127.14.14.131:44249/ using document root <none> and password file <none>
I20260928 22:57:35.359979 1405 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:35.360044 1405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:35.360186 1405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:35.361217 1405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2"
format_stamp: "Formatted at 2026-09-28 22:57:35 on dist-test-slave-kzfr"
server_key: "8173aaaa47aea47c93c6851d60532a55"
server_key_iv: "19915c6a6a5989a03ad0f4c119d9c3da"
server_key_version: "encryptionkey@0"
I20260928 22:57:35.361629 1405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2"
format_stamp: "Formatted at 2026-09-28 22:57:35 on dist-test-slave-kzfr"
server_key: "8173aaaa47aea47c93c6851d60532a55"
server_key_iv: "19915c6a6a5989a03ad0f4c119d9c3da"
server_key_version: "encryptionkey@0"
I20260928 22:57:35.363006 1405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:57:35.363693 1420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:35.363853 1405 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:35.363950 1405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2"
format_stamp: "Formatted at 2026-09-28 22:57:35 on dist-test-slave-kzfr"
server_key: "8173aaaa47aea47c93c6851d60532a55"
server_key_iv: "19915c6a6a5989a03ad0f4c119d9c3da"
server_key_version: "encryptionkey@0"
I20260928 22:57:35.364049 1405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:35.373245 1405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:35.375795 1405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:35.375964 1405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:35.376209 1405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:35.376530 1405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:35.376586 1405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:35.376629 1405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:35.376655 1405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:35.382885 1405 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:32919
I20260928 22:57:35.382928 1533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:32919 every 8 connection(s)
I20260928 22:57:35.383273 1405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:57:35 dist-test-slave-kzfr krb5kdc[1024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636255, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.131@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:57:35.389652 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 1405
I20260928 22:57:35.389788 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:57:35.389986 14394 external_mini_cluster.cc:1428] Setting key ab5980806d848e56b9ecaf374a79007f
I20260928 22:57:35.392932 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.384010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:56911 (local address 127.14.14.190:44163)
0928 22:57:35.384076 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:35.384078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:35.384288 (+ 210us) server_negotiation.cc:407] Connection header received
0928 22:57:35.384833 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:35.384834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:35.384847 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:35.384903 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:35.385958 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.386466 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.387531 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.387735 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.387822 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.390196 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:35.390198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:35.390198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:35.390205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:35.391710 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:35.392159 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:35.392160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:35.392160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:35.392210 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:35.392393 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:35.392393 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:35.392394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:35.392496 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:35.392545 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:35.392801 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 22:57:35.392865 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:57:35.393142 1536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.384014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44163 (local address 127.14.14.131:56911)
0928 22:57:35.384166 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:35.384203 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:35.384725 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:35.384988 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:35.384997 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:35.385632 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:35.385857 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.385862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.386567 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.386569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.387432 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.387434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.387830 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.387831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.387961 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.388463 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:57:35.388471 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:35.390093 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:35.391843 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:35.391849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:35.391851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:35.392087 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:35.392256 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:57:35.392257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:57:35.392257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:57:35.392348 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:57:35.392552 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:35.392557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:35.392679 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 22:57:35.392775 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 22:57:35.392904 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260928 22:57:35.393709 1534 heartbeater.cc:347] Connected to a master server at 127.14.14.190:44163
I20260928 22:57:35.393803 1534 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:35.393960 1534 heartbeater.cc:511] Master 127.14.14.190:44163 requested a full tablet report, sending...
I20260928 22:57:35.394295 1075 ts_manager.cc:194] Registered new tserver with Master: a15f407a40dd4a1c9fedbb1a5b23abb2 (127.14.14.131:32919)
I20260928 22:57:35.394711 1075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.14.131@KRBTEST.COM'} at 127.14.14.131:56911
I20260928 22:57:35.401986 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:57:35.409899 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.403362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47480 (local address 127.14.14.190:44163)
0928 22:57:35.403442 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:35.403444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:35.403590 (+ 146us) server_negotiation.cc:407] Connection header received
0928 22:57:35.403697 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:35.403698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:35.403710 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:35.403786 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:35.404379 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.404874 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.405639 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.405836 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.405918 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.407185 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:35.407186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:35.407187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:35.407194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:35.408645 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:35.409016 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:35.409017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:35.409018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:35.409076 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:35.409283 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:35.409285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:35.409285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:35.409443 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:35.409552 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:35.409709 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 22:57:35.409790 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 22:57:35.411437 1075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:57:35.413211 1075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:57:35.419615 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.415529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:32919 (local address 127.0.0.1:35364)
0928 22:57:35.415699 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:35.415740 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:35.415807 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:35.416054 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:35.416057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:35.416059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:35.416127 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.416129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.417267 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.417268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.418400 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.418403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.419300 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.419301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.419376 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.419389 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 22:57:35.419482 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 22:57:35.419542 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":60,"threads_started":1}
I20260928 22:57:35.419627 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.415705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.14.14.131:32919)
0928 22:57:35.415892 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:35.415897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:35.415907 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:57:35.415950 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:35.415952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:35.415969 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:35.416051 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:35.416254 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.417171 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.418498 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.419223 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.419302 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.419361 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:35.419500 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 22:57:35.419541 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":78,"threads_started":1}
I20260928 22:57:35.420288 1547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.415529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:45201 (local address 127.0.0.1:47920)
0928 22:57:35.415749 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:35.415763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:35.415820 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:35.416268 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:35.416269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:35.416271 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:35.416334 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.416335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.417902 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.417903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.419022 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.419024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.420029 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.420030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.420105 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.420109 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:57:35.420145 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 22:57:35.420190 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":58,"thread_start_us":68,"threads_started":1}
I20260928 22:57:35.420305 1550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.416121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.14.14.130:39425)
0928 22:57:35.416503 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:35.416507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:35.416517 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:57:35.416556 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:35.416558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:35.416576 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:35.416663 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:35.416783 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.417958 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.419168 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.419928 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.420031 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.420096 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:35.420177 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 22:57:35.420218 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":95,"threads_started":1}
I20260928 22:57:35.420542 1545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.415608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:39425 (local address 127.0.0.1:40176)
0928 22:57:35.415907 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:35.415916 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:35.415953 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:35.416670 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:35.416671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:35.416673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:35.416720 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.416721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.418061 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.418062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.419040 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.419043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.420030 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.420031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.420102 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.420106 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:57:35.420164 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:57:35.420204 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":98,"threads_started":1}
I20260928 22:57:35.420527 1549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.415869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47920 (local address 127.14.14.129:45201)
0928 22:57:35.416068 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:35.416074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:35.416086 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:57:35.416130 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:35.416131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:35.416149 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:35.416421 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:35.416507 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.417783 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.419115 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.419928 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.420257 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.420326 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:35.420354 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 22:57:35.420403 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":124,"threads_started":1}
I20260928 22:57:35.420537 1468 tablet_service.cc:1467] Processing CreateTablet for tablet 280a098df4564bb7bf7b8533b1231a35 (DEFAULT_TABLE table=test-table [id=bec2555df5854ab8993e4dd33be9fd6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:35.420933 1468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 280a098df4564bb7bf7b8533b1231a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:35.421183 1200 tablet_service.cc:1467] Processing CreateTablet for tablet 280a098df4564bb7bf7b8533b1231a35 (DEFAULT_TABLE table=test-table [id=bec2555df5854ab8993e4dd33be9fd6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:35.421500 1200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 280a098df4564bb7bf7b8533b1231a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:35.422147 1334 tablet_service.cc:1467] Processing CreateTablet for tablet 280a098df4564bb7bf7b8533b1231a35 (DEFAULT_TABLE table=test-table [id=bec2555df5854ab8993e4dd33be9fd6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:35.422433 1334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 280a098df4564bb7bf7b8533b1231a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:35.424571 1551 tablet_bootstrap.cc:492] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2: Bootstrap starting.
I20260928 22:57:35.425076 1552 tablet_bootstrap.cc:492] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894: Bootstrap starting.
I20260928 22:57:35.425316 1551 tablet_bootstrap.cc:654] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:35.425731 1552 tablet_bootstrap.cc:654] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:35.425760 1553 tablet_bootstrap.cc:492] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0: Bootstrap starting.
I20260928 22:57:35.425895 1551 log.cc:824] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:35.426308 1552 log.cc:824] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:35.426404 1553 tablet_bootstrap.cc:654] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:35.426632 1551 tablet_bootstrap.cc:492] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2: No bootstrap required, opened a new log
I20260928 22:57:35.426709 1551 ts_tablet_manager.cc:1397] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:57:35.426952 1553 log.cc:824] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:35.427047 1552 tablet_bootstrap.cc:492] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894: No bootstrap required, opened a new log
I20260928 22:57:35.427103 1552 ts_tablet_manager.cc:1397] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:57:35.427587 1553 tablet_bootstrap.cc:492] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0: No bootstrap required, opened a new log
I20260928 22:57:35.427670 1553 ts_tablet_manager.cc:1397] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:57:35.428721 1551 raft_consensus.cc:348] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494a6638978a4cb6ad42048d391de894" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45201 } } peers { permanent_uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 32919 } } peers { permanent_uuid: "9a0185d8746741329503c060ad30fda0" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 39425 } }
I20260928 22:57:35.428926 1551 raft_consensus.cc:374] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:35.428975 1551 raft_consensus.cc:729] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15f407a40dd4a1c9fedbb1a5b23abb2, State: Initialized, Role: FOLLOWER
I20260928 22:57:35.429085 1551 consensus_queue.cc:260] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494a6638978a4cb6ad42048d391de894" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45201 } } peers { permanent_uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 32919 } } peers { permanent_uuid: "9a0185d8746741329503c060ad30fda0" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 39425 } }
I20260928 22:57:35.429023 1552 raft_consensus.cc:348] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494a6638978a4cb6ad42048d391de894" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45201 } } peers { permanent_uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 32919 } } peers { permanent_uuid: "9a0185d8746741329503c060ad30fda0" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 39425 } }
I20260928 22:57:35.429168 1552 raft_consensus.cc:374] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:35.429206 1552 raft_consensus.cc:729] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494a6638978a4cb6ad42048d391de894, State: Initialized, Role: FOLLOWER
I20260928 22:57:35.429152 1553 raft_consensus.cc:348] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494a6638978a4cb6ad42048d391de894" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45201 } } peers { permanent_uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 32919 } } peers { permanent_uuid: "9a0185d8746741329503c060ad30fda0" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 39425 } }
I20260928 22:57:35.429345 1553 raft_consensus.cc:374] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:35.429359 1551 ts_tablet_manager.cc:1428] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:57:35.429392 1553 raft_consensus.cc:729] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a0185d8746741329503c060ad30fda0, State: Initialized, Role: FOLLOWER
I20260928 22:57:35.429360 1552 consensus_queue.cc:260] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494a6638978a4cb6ad42048d391de894" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45201 } } peers { permanent_uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 32919 } } peers { permanent_uuid: "9a0185d8746741329503c060ad30fda0" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 39425 } }
I20260928 22:57:35.429469 1534 heartbeater.cc:503] Master 127.14.14.190:44163 was elected leader, sending a full tablet report...
I20260928 22:57:35.429517 1553 consensus_queue.cc:260] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494a6638978a4cb6ad42048d391de894" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45201 } } peers { permanent_uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 32919 } } peers { permanent_uuid: "9a0185d8746741329503c060ad30fda0" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 39425 } }
I20260928 22:57:35.429648 1552 ts_tablet_manager.cc:1428] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 22:57:35.429765 1266 heartbeater.cc:503] Master 127.14.14.190:44163 was elected leader, sending a full tablet report...
I20260928 22:57:35.429800 1553 ts_tablet_manager.cc:1428] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:57:35.429804 1400 heartbeater.cc:503] Master 127.14.14.190:44163 was elected leader, sending a full tablet report...
W20260928 22:57:35.480302 1401 tablet.cc:2369] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:57:35.562611 1267 tablet.cc:2369] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:57:35.633931 1535 tablet.cc:2369] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:57:35.646312 1557 raft_consensus.cc:482] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:57:35.646436 1557 raft_consensus.cc:504] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494a6638978a4cb6ad42048d391de894" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45201 } } peers { permanent_uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 32919 } } peers { permanent_uuid: "9a0185d8746741329503c060ad30fda0" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 39425 } }
I20260928 22:57:35.646772 1557 leader_election.cc:288] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 494a6638978a4cb6ad42048d391de894 (127.14.14.129:45201), 9a0185d8746741329503c060ad30fda0 (127.14.14.130:39425)
I20260928 22:57:35.650755 1549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.647060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:51841 (local address 127.14.14.129:45201)
0928 22:57:35.647106 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:35.647108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:35.647312 (+ 204us) server_negotiation.cc:407] Connection header received
0928 22:57:35.647345 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:35.647346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:35.647355 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:35.647419 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:35.647621 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.648302 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.649670 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.650219 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.650308 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.650317 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:35.650649 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 22:57:35.650689 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:57:35.650755 1550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.647078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:47561 (local address 127.14.14.130:39425)
0928 22:57:35.647125 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:35.647127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:35.647137 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:57:35.647167 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:35.647168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:35.647180 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:35.647248 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:35.647430 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.648210 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.649653 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.650249 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.650316 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.650323 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:35.650641 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 22:57:35.650686 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:57:35.650754 1536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.646974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:39425 (local address 127.14.14.131:47561)
0928 22:57:35.647034 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:35.647046 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:35.647096 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:35.647268 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:35.647269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:35.647271 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:35.647345 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.647347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.648300 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.648301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.649531 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.649534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.650492 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.650493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.650564 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.650567 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:57:35.650620 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:57:35.650660 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:57:35.650760 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.646981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:45201 (local address 127.14.14.131:51841)
0928 22:57:35.647239 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:35.647248 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:35.647297 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:35.647500 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:35.647501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:35.647502 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:35.647551 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.647552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.648375 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.648376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.649531 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:35.649534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.650453 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.650454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:35.650538 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.650544 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:57:35.650609 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 22:57:35.650658 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":84,"thread_start_us":123,"threads_started":1}
I20260928 22:57:35.651185 1354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280a098df4564bb7bf7b8533b1231a35" candidate_uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a0185d8746741329503c060ad30fda0" is_pre_election: true
I20260928 22:57:35.651363 1354 raft_consensus.cc:2454] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a15f407a40dd4a1c9fedbb1a5b23abb2 in term 0.
I20260928 22:57:35.651389 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280a098df4564bb7bf7b8533b1231a35" candidate_uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494a6638978a4cb6ad42048d391de894" is_pre_election: true
I20260928 22:57:35.651561 1220 raft_consensus.cc:2454] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a15f407a40dd4a1c9fedbb1a5b23abb2 in term 0.
I20260928 22:57:35.651640 1424 leader_election.cc:302] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a0185d8746741329503c060ad30fda0, a15f407a40dd4a1c9fedbb1a5b23abb2; no voters:
I20260928 22:57:35.651770 1557 raft_consensus.cc:2781] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:57:35.651824 1557 raft_consensus.cc:482] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:57:35.651861 1557 raft_consensus.cc:3037] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:35.652565 1557 raft_consensus.cc:504] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494a6638978a4cb6ad42048d391de894" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45201 } } peers { permanent_uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 32919 } } peers { permanent_uuid: "9a0185d8746741329503c060ad30fda0" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 39425 } }
I20260928 22:57:35.652706 1557 leader_election.cc:288] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [CANDIDATE]: Term 1 election: Requested vote from peers 494a6638978a4cb6ad42048d391de894 (127.14.14.129:45201), 9a0185d8746741329503c060ad30fda0 (127.14.14.130:39425)
I20260928 22:57:35.652853 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280a098df4564bb7bf7b8533b1231a35" candidate_uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494a6638978a4cb6ad42048d391de894"
I20260928 22:57:35.652846 1354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280a098df4564bb7bf7b8533b1231a35" candidate_uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a0185d8746741329503c060ad30fda0"
I20260928 22:57:35.652946 1220 raft_consensus.cc:3037] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:35.652951 1354 raft_consensus.cc:3037] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:35.653733 1220 raft_consensus.cc:2454] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a15f407a40dd4a1c9fedbb1a5b23abb2 in term 1.
I20260928 22:57:35.653743 1354 raft_consensus.cc:2454] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a15f407a40dd4a1c9fedbb1a5b23abb2 in term 1.
I20260928 22:57:35.653896 1422 leader_election.cc:302] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 494a6638978a4cb6ad42048d391de894, a15f407a40dd4a1c9fedbb1a5b23abb2; no voters:
I20260928 22:57:35.654019 1557 raft_consensus.cc:2781] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:35.654171 1557 raft_consensus.cc:686] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [term 1 LEADER]: Becoming Leader. State: Replica: a15f407a40dd4a1c9fedbb1a5b23abb2, State: Running, Role: LEADER
I20260928 22:57:35.654263 1557 consensus_queue.cc:237] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494a6638978a4cb6ad42048d391de894" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45201 } } peers { permanent_uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 32919 } } peers { permanent_uuid: "9a0185d8746741329503c060ad30fda0" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 39425 } }
I20260928 22:57:35.655087 1073 catalog_manager.cc:5495] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a15f407a40dd4a1c9fedbb1a5b23abb2 (127.14.14.131). New cstate: current_term: 1 leader_uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494a6638978a4cb6ad42048d391de894" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15f407a40dd4a1c9fedbb1a5b23abb2" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0185d8746741329503c060ad30fda0" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 39425 } health_report { overall_health: UNKNOWN } } }
I20260928 22:57:35.702159 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.699804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.14.14.131:32919)
0928 22:57:35.699864 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:35.699866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:35.699874 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:57:35.699927 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:35.699928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:35.699945 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:35.700017 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:57:35.700110 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.700572 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.701309 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.701550 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.701652 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.701776 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:57:35.701913 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:57:35.701976 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:35.701999 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 22:57:35.702017 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:57:35.704025 1354 raft_consensus.cc:1261] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0 [term 1 FOLLOWER]: Refusing update from remote peer a15f407a40dd4a1c9fedbb1a5b23abb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:57:35.704025 1220 raft_consensus.cc:1261] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894 [term 1 FOLLOWER]: Refusing update from remote peer a15f407a40dd4a1c9fedbb1a5b23abb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:57:35.704388 1561 consensus_queue.cc:1035] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0185d8746741329503c060ad30fda0" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:57:35.704571 1557 consensus_queue.cc:1035] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "494a6638978a4cb6ad42048d391de894" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:57:35.709405 1074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:57:35.709491 1074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:57:35.710227 1074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84fd0731a2a94becbd897b78ba9f3103: Sending DeleteTablet for 3 replicas of tablet 280a098df4564bb7bf7b8533b1231a35
I20260928 22:57:35.710531 1200 tablet_service.cc:1514] Processing DeleteTablet for tablet 280a098df4564bb7bf7b8533b1231a35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:35 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:47920
I20260928 22:57:35.710561 1334 tablet_service.cc:1514] Processing DeleteTablet for tablet 280a098df4564bb7bf7b8533b1231a35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:35 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:40176
I20260928 22:57:35.710655 1468 tablet_service.cc:1514] Processing DeleteTablet for tablet 280a098df4564bb7bf7b8533b1231a35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:35 UTC) from {username='kudu', principal='kudu/127.14.14.190@KRBTEST.COM'} at 127.0.0.1:35364
I20260928 22:57:35.710827 1575 tablet_replica.cc:330] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894: stopping tablet replica
I20260928 22:57:35.710908 1576 tablet_replica.cc:330] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2: stopping tablet replica
I20260928 22:57:35.710978 1575 raft_consensus.cc:2229] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:57:35.711028 1576 raft_consensus.cc:2229] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:57:35.711156 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 1137
I20260928 22:57:35.711165 1575 raft_consensus.cc:2258] T 280a098df4564bb7bf7b8533b1231a35 P 494a6638978a4cb6ad42048d391de894 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:57:35.711179 1576 raft_consensus.cc:2258] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:57:35.711323 1574 tablet_replica.cc:330] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0: stopping tablet replica
I20260928 22:57:35.711400 1574 raft_consensus.cc:2229] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:57:35.711477 1574 raft_consensus.cc:2258] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:57:35.712479 1574 ts_tablet_manager.cc:1905] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:57:35.712479 1576 ts_tablet_manager.cc:1905] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:57:35.714526 1576 ts_tablet_manager.cc:1918] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:57:35.714526 1574 ts_tablet_manager.cc:1918] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:57:35.714594 1576 log.cc:1196] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/280a098df4564bb7bf7b8533b1231a35
I20260928 22:57:35.714594 1574 log.cc:1196] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/280a098df4564bb7bf7b8533b1231a35
I20260928 22:57:35.714879 1574 ts_tablet_manager.cc:1939] T 280a098df4564bb7bf7b8533b1231a35 P 9a0185d8746741329503c060ad30fda0: Deleting consensus metadata
I20260928 22:57:35.714879 1576 ts_tablet_manager.cc:1939] T 280a098df4564bb7bf7b8533b1231a35 P a15f407a40dd4a1c9fedbb1a5b23abb2: Deleting consensus metadata
I20260928 22:57:35.715246 1062 catalog_manager.cc:4841] TS 9a0185d8746741329503c060ad30fda0 (127.14.14.130:39425): tablet 280a098df4564bb7bf7b8533b1231a35 (table test-table [id=bec2555df5854ab8993e4dd33be9fd6a]) successfully deleted
I20260928 22:57:35.715325 1054 catalog_manager.cc:4841] TS a15f407a40dd4a1c9fedbb1a5b23abb2 (127.14.14.131:32919): tablet 280a098df4564bb7bf7b8533b1231a35 (table test-table [id=bec2555df5854ab8993e4dd33be9fd6a]) successfully deleted
W20260928 22:57:35.718245 1055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:45201 (error 108)
I20260928 22:57:35.718796 1545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.718651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:45201 (local address 127.0.0.1:47936)
0928 22:57:35.718703 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:35.718743 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:45201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 22:57:35.718842 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 1271
W20260928 22:57:35.720086 1055 catalog_manager.cc:4584] TS 494a6638978a4cb6ad42048d391de894 (127.14.14.129:45201): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 280a098df4564bb7bf7b8533b1231a35: Network error: Client connection negotiation failed: client connection to 127.14.14.129:45201: connect: Connection refused (error 111)
I20260928 22:57:35.725208 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 1405
I20260928 22:57:35.729552 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 1038
2026-09-28T22:57:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2113 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T22:57:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:35Z Disabled control of system clock
I20260928 22:57:35.756520 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:36579
--webserver_interface=127.14.14.190
--webserver_port=0
--builtin_ntp_servers=127.14.14.148:34663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:36579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:57:35.838821 1581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:35.839071 1581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:35.839105 1581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:35.840222 1581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:35.840286 1581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:35.840317 1581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:35.840348 1581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:35.840377 1581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:35.841765 1581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:34663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:36579
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:36579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:35.842023 1581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:35.842298 1581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:35.846961 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:35.847028 1586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:35.846913 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:35.847141 1581 server_base.cc:1034] running on GCE node
I20260928 22:57:35.847431 1581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:35.847702 1581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:35.848839 1581 hybrid_clock.cc:648] HybridClock initialized: now 1790636255848822 us; error 36 us; skew 500 ppm
I20260928 22:57:35.850119 1581 webserver.cc:460] Webserver started at http://127.14.14.190:38009/ using document root <none> and password file <none>
I20260928 22:57:35.850380 1581 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:35.850426 1581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:35.850580 1581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:35.851622 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "8815a8e2560644a9979026da3b64f3ee"
format_stamp: "Formatted at 2026-09-28 22:57:35 on dist-test-slave-kzfr"
server_key: "972911f0306a509237494da652c5c76f"
server_key_iv: "cd2a6cdb69d4eb893206e5fe4da7229c"
server_key_version: "encryptionkey@0"
I20260928 22:57:35.851984 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "8815a8e2560644a9979026da3b64f3ee"
format_stamp: "Formatted at 2026-09-28 22:57:35 on dist-test-slave-kzfr"
server_key: "972911f0306a509237494da652c5c76f"
server_key_iv: "cd2a6cdb69d4eb893206e5fe4da7229c"
server_key_version: "encryptionkey@0"
I20260928 22:57:35.853237 1581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:57:35.853957 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:35.854136 1581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:35.854233 1581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "8815a8e2560644a9979026da3b64f3ee"
format_stamp: "Formatted at 2026-09-28 22:57:35 on dist-test-slave-kzfr"
server_key: "972911f0306a509237494da652c5c76f"
server_key_iv: "cd2a6cdb69d4eb893206e5fe4da7229c"
server_key_version: "encryptionkey@0"
I20260928 22:57:35.854308 1581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:35.864581 1581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:35.864864 1581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:35.865008 1581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:35.868711 1581 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:36579
I20260928 22:57:35.868767 1647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:36579 every 8 connection(s)
I20260928 22:57:35.869151 1581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:35.870221 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:35.870451 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 1581
I20260928 22:57:35.870574 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:35.870764 14394 external_mini_cluster.cc:1428] Setting key bd033bda1a407ab81d63678c78efed45
I20260928 22:57:35.872895 1648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee: Bootstrap starting.
I20260928 22:57:35.873518 1648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:35.873859 1648 log.cc:824] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:35.874547 1648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee: No bootstrap required, opened a new log
I20260928 22:57:35.876153 1651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:35.871288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56012 (local address 127.14.14.190:36579)
0928 22:57:35.871492 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:35.871500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:35.871524 (+ 24us) server_negotiation.cc:407] Connection header received
0928 22:57:35.871932 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:35.871938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:35.871995 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:35.872109 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:35.872345 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.873255 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.874335 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:35.874778 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:35.874868 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:35.875507 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:35.875511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:35.875511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:35.875683 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:35.875714 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:35.875724 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:35.875774 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:35.875870 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 22:57:35.875926 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260928 22:57:35.876730 1648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8815a8e2560644a9979026da3b64f3ee" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36579 } }
I20260928 22:57:35.876878 1648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:35.876940 1648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8815a8e2560644a9979026da3b64f3ee, State: Initialized, Role: FOLLOWER
I20260928 22:57:35.877081 1648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8815a8e2560644a9979026da3b64f3ee" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36579 } }
I20260928 22:57:35.877157 1648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:35.877199 1648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:35.877246 1648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:35.877822 1648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8815a8e2560644a9979026da3b64f3ee" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36579 } }
I20260928 22:57:35.877933 1648 leader_election.cc:302] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8815a8e2560644a9979026da3b64f3ee; no voters:
I20260928 22:57:35.878129 1648 leader_election.cc:288] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:35.878167 1653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:35.878306 1653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [term 1 LEADER]: Becoming Leader. State: Replica: 8815a8e2560644a9979026da3b64f3ee, State: Running, Role: LEADER
I20260928 22:57:35.878403 1648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:35.878432 1653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8815a8e2560644a9979026da3b64f3ee" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36579 } }
I20260928 22:57:35.878826 1654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8815a8e2560644a9979026da3b64f3ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8815a8e2560644a9979026da3b64f3ee" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36579 } } }
I20260928 22:57:35.878962 1654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:35.878841 1655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8815a8e2560644a9979026da3b64f3ee. Latest consensus state: current_term: 1 leader_uuid: "8815a8e2560644a9979026da3b64f3ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8815a8e2560644a9979026da3b64f3ee" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 36579 } } }
I20260928 22:57:35.879076 1655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:35.879426 1663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:35.879911 1663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:35.881338 1663 catalog_manager.cc:1347] Generated new cluster ID: 97af3c566e0946c2b5fb4829e2c32d67
I20260928 22:57:35.881387 1663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:35.905607 1663 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:35.906153 1663 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:35.912320 1663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee: Generated new TSK 0
I20260928 22:57:35.912506 1663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:35.920286 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.129:0
--local_ip_for_outbound_sockets=127.14.14.129
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36579
--builtin_ntp_servers=127.14.14.148:34663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:57:36.026564 1672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:36.026755 1672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:36.026777 1672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:36.027885 1672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:36.027925 1672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:36.027951 1672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.129
I20260928 22:57:36.029260 1672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:34663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.14.129
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.129
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:36.029505 1672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:36.029820 1672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:36.034962 1677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:36.035028 1678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:36.035044 1672 server_base.cc:1034] running on GCE node
W20260928 22:57:36.034964 1680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:36.035424 1672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:36.035687 1672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:36.036847 1672 hybrid_clock.cc:648] HybridClock initialized: now 1790636256036845 us; error 37 us; skew 500 ppm
I20260928 22:57:36.038090 1672 webserver.cc:460] Webserver started at http://127.14.14.129:40623/ using document root <none> and password file <none>
I20260928 22:57:36.038337 1672 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:36.038403 1672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:36.038542 1672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:36.039594 1672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/data/instance:
uuid: "dc56d3b9199647d39d0352a17bdc1861"
format_stamp: "Formatted at 2026-09-28 22:57:36 on dist-test-slave-kzfr"
server_key: "a0cee3609b1507dfb05885adf6eff891"
server_key_iv: "a374588faf27c164511f550ad850ace9"
server_key_version: "encryptionkey@0"
I20260928 22:57:36.039939 1672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance:
uuid: "dc56d3b9199647d39d0352a17bdc1861"
format_stamp: "Formatted at 2026-09-28 22:57:36 on dist-test-slave-kzfr"
server_key: "a0cee3609b1507dfb05885adf6eff891"
server_key_iv: "a374588faf27c164511f550ad850ace9"
server_key_version: "encryptionkey@0"
I20260928 22:57:36.041224 1672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:57:36.041990 1686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:36.042155 1672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:36.042246 1672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/wal
uuid: "dc56d3b9199647d39d0352a17bdc1861"
format_stamp: "Formatted at 2026-09-28 22:57:36 on dist-test-slave-kzfr"
server_key: "a0cee3609b1507dfb05885adf6eff891"
server_key_iv: "a374588faf27c164511f550ad850ace9"
server_key_version: "encryptionkey@0"
I20260928 22:57:36.042323 1672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:36.051259 1672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:36.051550 1672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:36.051692 1672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:36.051903 1672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:36.052223 1672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:36.052274 1672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:36.052315 1672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:36.052345 1672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:36.058493 1672 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.129:44451
I20260928 22:57:36.058550 1799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.129:44451 every 8 connection(s)
I20260928 22:57:36.058861 1672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/data/info.pb
I20260928 22:57:36.064158 1651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.059553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.129:52285 (local address 127.14.14.190:36579)
0928 22:57:36.059616 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:36.059618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:36.059850 (+ 232us) server_negotiation.cc:407] Connection header received
0928 22:57:36.060389 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:36.060391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:36.060403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:36.060473 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:36.061151 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.061740 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.062735 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.062941 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.063015 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.063856 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:36.063858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:36.063858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:36.063863 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:36.063877 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:36.063881 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:36.063944 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:36.064038 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 22:57:36.064076 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:57:36.064380 1802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.059571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:36579 (local address 127.14.14.129:52285)
0928 22:57:36.059728 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:36.059761 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:36.060283 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:36.060549 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:36.060553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:36.060560 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:36.061052 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.061056 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.061830 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.061832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.062642 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.062644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.063022 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.063024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.063161 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.063712 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:57:36.063712 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:36.063741 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:57:36.063763 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:36.063946 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:36.063952 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:36.063956 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:57:36.064039 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:57:36.064151 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260928 22:57:36.064926 1800 heartbeater.cc:347] Connected to a master server at 127.14.14.190:36579
I20260928 22:57:36.065006 1800 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:36.065213 1800 heartbeater.cc:511] Master 127.14.14.190:36579 requested a full tablet report, sending...
I20260928 22:57:36.065783 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 1672
I20260928 22:57:36.065856 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-0/wal/instance
I20260928 22:57:36.065896 1612 ts_manager.cc:194] Registered new tserver with Master: dc56d3b9199647d39d0352a17bdc1861 (127.14.14.129:44451)
I20260928 22:57:36.066087 14394 external_mini_cluster.cc:1428] Setting key 8ae4c94ab13f2df59a72af87dcc5d2bb
I20260928 22:57:36.066699 1612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.14.129:52285
I20260928 22:57:36.067184 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.130:0
--local_ip_for_outbound_sockets=127.14.14.130
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36579
--builtin_ntp_servers=127.14.14.148:34663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:57:36.151834 1803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:36.152024 1803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:36.152045 1803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:36.153187 1803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:36.153227 1803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:36.153275 1803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.130
I20260928 22:57:36.154680 1803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:34663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.14.130
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.130
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:36.154913 1803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:36.155205 1803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:36.160321 1808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:36.160322 1809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:36.160346 1811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:36.160526 1803 server_base.cc:1034] running on GCE node
I20260928 22:57:36.160715 1803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:36.160946 1803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:36.162132 1803 hybrid_clock.cc:648] HybridClock initialized: now 1790636256162115 us; error 34 us; skew 500 ppm
I20260928 22:57:36.163609 1803 webserver.cc:460] Webserver started at http://127.14.14.130:36745/ using document root <none> and password file <none>
I20260928 22:57:36.163884 1803 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:36.163959 1803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:36.164103 1803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:36.165232 1803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/data/instance:
uuid: "d8d6cde850854019855d524224f1fdbe"
format_stamp: "Formatted at 2026-09-28 22:57:36 on dist-test-slave-kzfr"
server_key: "8fbe104ce11743cfbdd1da9fe685960e"
server_key_iv: "9d28ff3b9c45a138194604671cb3d785"
server_key_version: "encryptionkey@0"
I20260928 22:57:36.165654 1803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance:
uuid: "d8d6cde850854019855d524224f1fdbe"
format_stamp: "Formatted at 2026-09-28 22:57:36 on dist-test-slave-kzfr"
server_key: "8fbe104ce11743cfbdd1da9fe685960e"
server_key_iv: "9d28ff3b9c45a138194604671cb3d785"
server_key_version: "encryptionkey@0"
I20260928 22:57:36.167078 1803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:36.167872 1817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:36.168021 1803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:36.168110 1803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/wal
uuid: "d8d6cde850854019855d524224f1fdbe"
format_stamp: "Formatted at 2026-09-28 22:57:36 on dist-test-slave-kzfr"
server_key: "8fbe104ce11743cfbdd1da9fe685960e"
server_key_iv: "9d28ff3b9c45a138194604671cb3d785"
server_key_version: "encryptionkey@0"
I20260928 22:57:36.168188 1803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:36.207561 1803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:36.207921 1803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:36.208079 1803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:36.208331 1803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:36.208673 1803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:36.208727 1803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:36.208762 1803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:36.208806 1803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:36.214546 1803 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.130:43691
I20260928 22:57:36.214593 1930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.130:43691 every 8 connection(s)
I20260928 22:57:36.214932 1803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/data/info.pb
I20260928 22:57:36.220121 1651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.215678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:58093 (local address 127.14.14.190:36579)
0928 22:57:36.215742 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:36.215744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:36.215993 (+ 249us) server_negotiation.cc:407] Connection header received
0928 22:57:36.216460 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:36.216461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:36.216470 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:36.216532 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:36.217176 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.217674 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.218881 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.219065 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.219122 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.219894 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:36.219896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:36.219896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:36.219900 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:36.219915 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:36.219918 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:36.219961 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:36.220039 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 22:57:36.220062 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 22:57:36.220306 1933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.215662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:36579 (local address 127.14.14.130:58093)
0928 22:57:36.215856 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:36.215898 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:36.216384 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:36.216586 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:36.216589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:36.216597 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:36.217106 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.217110 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.217752 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.217753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.218802 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.218805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.219140 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.219141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.219251 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.219763 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:57:36.219763 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:36.219794 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:57:36.219817 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:36.219970 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:36.219974 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:36.219976 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:57:36.220030 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 22:57:36.220109 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260928 22:57:36.220937 1931 heartbeater.cc:347] Connected to a master server at 127.14.14.190:36579
I20260928 22:57:36.221045 1931 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:36.221232 1931 heartbeater.cc:511] Master 127.14.14.190:36579 requested a full tablet report, sending...
I20260928 22:57:36.221604 1612 ts_manager.cc:194] Registered new tserver with Master: d8d6cde850854019855d524224f1fdbe (127.14.14.130:43691)
I20260928 22:57:36.221964 1612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.14.130:58093
I20260928 22:57:36.222517 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 1803
I20260928 22:57:36.222623 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/wal/instance
I20260928 22:57:36.222818 14394 external_mini_cluster.cc:1428] Setting key a5943a66cb3d69e597fbf0b5ccafbc24
I20260928 22:57:36.223812 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.14.131:0
--local_ip_for_outbound_sockets=127.14.14.131
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36579
--builtin_ntp_servers=127.14.14.148:34663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:57:36.312260 1934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:36.312470 1934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:36.312498 1934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:36.313689 1934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:36.313727 1934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:36.313757 1934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.14.131
I20260928 22:57:36.315591 1934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:34663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.14.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.14.131
--webserver_port=0
--tserver_master_addrs=127.14.14.190:36579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.14.131
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:36.315832 1934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:36.316154 1934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:36.321326 1939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:36.321341 1934 server_base.cc:1034] running on GCE node
W20260928 22:57:36.321307 1942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:36.321308 1940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:36.321834 1934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:36.322070 1934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:36.323231 1934 hybrid_clock.cc:648] HybridClock initialized: now 1790636256323211 us; error 36 us; skew 500 ppm
I20260928 22:57:36.324448 1934 webserver.cc:460] Webserver started at http://127.14.14.131:33207/ using document root <none> and password file <none>
I20260928 22:57:36.324694 1934 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:36.324764 1934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:36.324915 1934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:36.326026 1934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/data/instance:
uuid: "49fe880a7cc246e1bb89926ccbff07cc"
format_stamp: "Formatted at 2026-09-28 22:57:36 on dist-test-slave-kzfr"
server_key: "f99400be4671ada1464c25f462721633"
server_key_iv: "144eb2c4fba18fe2c0c3df195daf02e9"
server_key_version: "encryptionkey@0"
I20260928 22:57:36.326378 1934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance:
uuid: "49fe880a7cc246e1bb89926ccbff07cc"
format_stamp: "Formatted at 2026-09-28 22:57:36 on dist-test-slave-kzfr"
server_key: "f99400be4671ada1464c25f462721633"
server_key_iv: "144eb2c4fba18fe2c0c3df195daf02e9"
server_key_version: "encryptionkey@0"
I20260928 22:57:36.327777 1934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:57:36.328601 1948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:36.328791 1934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:36.328891 1934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/wal
uuid: "49fe880a7cc246e1bb89926ccbff07cc"
format_stamp: "Formatted at 2026-09-28 22:57:36 on dist-test-slave-kzfr"
server_key: "f99400be4671ada1464c25f462721633"
server_key_iv: "144eb2c4fba18fe2c0c3df195daf02e9"
server_key_version: "encryptionkey@0"
I20260928 22:57:36.328969 1934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:36.348977 1934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:36.349287 1934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:36.349440 1934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:36.349728 1934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:57:36.350087 1934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:57:36.350147 1934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:36.350193 1934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:57:36.350226 1934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:36.356251 1934 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.131:40519
I20260928 22:57:36.356288 2061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.131:40519 every 8 connection(s)
I20260928 22:57:36.356727 1934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/data/info.pb
I20260928 22:57:36.358883 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 1934
I20260928 22:57:36.358968 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/wal/instance
I20260928 22:57:36.359169 14394 external_mini_cluster.cc:1428] Setting key d3be2a946c5b878b6c660fde48583c19
I20260928 22:57:36.362711 1651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.357482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.131:43303 (local address 127.14.14.190:36579)
0928 22:57:36.357566 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:36.357568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:36.357842 (+ 274us) server_negotiation.cc:407] Connection header received
0928 22:57:36.358401 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:36.358402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:36.358414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:36.358478 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:36.359318 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.360138 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.361193 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.361435 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.361525 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.362322 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:36.362323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:36.362324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:36.362329 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:36.362345 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:36.362349 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:36.362407 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:36.362599 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 22:57:36.362627 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 22:57:36.362946 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.357489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:36579 (local address 127.14.14.131:43303)
0928 22:57:36.357721 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:36.357755 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:36.358289 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:36.358558 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:36.358562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:36.358570 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:57:36.359140 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.359146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.360228 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.360229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.361082 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.361084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.361531 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.361532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.361703 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.362161 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:57:36.362161 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:57:36.362190 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:57:36.362212 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:57:36.362416 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:57:36.362421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:57:36.362424 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:57:36.362599 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 22:57:36.362705 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":120,"threads_started":1}
I20260928 22:57:36.363377 2062 heartbeater.cc:347] Connected to a master server at 127.14.14.190:36579
I20260928 22:57:36.363451 2062 heartbeater.cc:464] Registering TS with master...
I20260928 22:57:36.363648 2062 heartbeater.cc:511] Master 127.14.14.190:36579 requested a full tablet report, sending...
I20260928 22:57:36.364042 1612 ts_manager.cc:194] Registered new tserver with Master: 49fe880a7cc246e1bb89926ccbff07cc (127.14.14.131:40519)
I20260928 22:57:36.364414 1612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.14.131:43303
I20260928 22:57:36.371002 14394 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:57:36.375535 1651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.372454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.14.14.190:36579)
0928 22:57:36.372521 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:36.372523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:36.372709 (+ 186us) server_negotiation.cc:407] Connection header received
0928 22:57:36.372775 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:36.372776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:36.372787 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:36.372844 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:36.373027 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.373513 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.374395 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.374607 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.374685 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.375310 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:36.375311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:36.375311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:36.375316 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:36.375329 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:36.375333 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:36.375380 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:36.375428 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 22:57:36.375449 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:57:36.377046 1612 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:57:36.378712 1612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:57:36.385691 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.381059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:44451 (local address 127.0.0.1:44996)
0928 22:57:36.381337 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:36.381346 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:36.381387 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:36.381885 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:36.381889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:36.381891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:36.381967 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.381969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.383123 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.383124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.384396 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.384399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.385344 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.385345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.385424 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.385438 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:57:36.385535 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 22:57:36.385611 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":49,"thread_start_us":58,"threads_started":1}
I20260928 22:57:36.385877 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.381230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.14.14.131:40519)
0928 22:57:36.382004 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:36.382008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:36.382017 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:57:36.382054 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:36.382056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:36.382071 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:36.382144 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:36.382286 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.382953 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.384384 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.385225 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.385309 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.385414 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:36.385734 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 22:57:36.385776 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":93,"threads_started":1}
I20260928 22:57:36.385932 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.381048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:40519 (local address 127.0.0.1:52002)
0928 22:57:36.381220 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:36.381262 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:36.381326 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:36.382164 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:36.382165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:36.382168 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:36.382212 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.382214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.383046 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.383047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.384288 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.384290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.385451 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.385452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.385540 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.385543 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:57:36.385634 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 22:57:36.385686 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":63,"thread_start_us":68,"threads_started":1}
I20260928 22:57:36.386296 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.381170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.14.14.129:44451)
0928 22:57:36.381713 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:36.381718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:36.381727 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:57:36.381768 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:36.381770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:36.381786 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:36.381872 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:36.382070 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.383037 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.384495 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.385239 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.385343 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.385427 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:36.385552 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 22:57:36.385726 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":81,"threads_started":1}
I20260928 22:57:36.386529 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.381254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.130:43691 (local address 127.0.0.1:35998)
0928 22:57:36.381479 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:36.381501 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:36.381561 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:36.383152 (+ 1591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:36.383153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:36.383155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:36.383201 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.383202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.384512 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.384513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.385182 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.385184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.386289 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.386290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.386369 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.386374 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:57:36.386430 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:57:36.386466 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":65,"threads_started":1}
I20260928 22:57:36.386791 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.381462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35998 (local address 127.14.14.130:43691)
0928 22:57:36.381689 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:36.381693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:36.381704 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:57:36.381768 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:36.381770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:36.381786 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:36.381872 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:36.383290 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.384425 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.385303 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.386055 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.386154 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.386233 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:36.386670 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 22:57:36.386711 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
I20260928 22:57:36.386835 1996 tablet_service.cc:1467] Processing CreateTablet for tablet 3e995d5022594fc6b97b57fa8830d6fa (DEFAULT_TABLE table=test-table [id=8fd55a5195544d37af17880971053475]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:36.387215 1996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e995d5022594fc6b97b57fa8830d6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:36.387158 1734 tablet_service.cc:1467] Processing CreateTablet for tablet 3e995d5022594fc6b97b57fa8830d6fa (DEFAULT_TABLE table=test-table [id=8fd55a5195544d37af17880971053475]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:36.387526 1734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e995d5022594fc6b97b57fa8830d6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:36.387477 1865 tablet_service.cc:1467] Processing CreateTablet for tablet 3e995d5022594fc6b97b57fa8830d6fa (DEFAULT_TABLE table=test-table [id=8fd55a5195544d37af17880971053475]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:57:36.387807 1865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e995d5022594fc6b97b57fa8830d6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:36.390487 2076 tablet_bootstrap.cc:492] T 3e995d5022594fc6b97b57fa8830d6fa P dc56d3b9199647d39d0352a17bdc1861: Bootstrap starting.
I20260928 22:57:36.390975 2077 tablet_bootstrap.cc:492] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc: Bootstrap starting.
I20260928 22:57:36.391037 2078 tablet_bootstrap.cc:492] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe: Bootstrap starting.
I20260928 22:57:36.391413 2076 tablet_bootstrap.cc:654] T 3e995d5022594fc6b97b57fa8830d6fa P dc56d3b9199647d39d0352a17bdc1861: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:36.391577 2077 tablet_bootstrap.cc:654] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:36.391733 2078 tablet_bootstrap.cc:654] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:36.391772 2076 log.cc:824] T 3e995d5022594fc6b97b57fa8830d6fa P dc56d3b9199647d39d0352a17bdc1861: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:36.391938 2077 log.cc:824] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:36.392066 2078 log.cc:824] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:36.392776 2077 tablet_bootstrap.cc:492] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc: No bootstrap required, opened a new log
I20260928 22:57:36.392881 2076 tablet_bootstrap.cc:492] T 3e995d5022594fc6b97b57fa8830d6fa P dc56d3b9199647d39d0352a17bdc1861: No bootstrap required, opened a new log
I20260928 22:57:36.392884 2077 ts_tablet_manager.cc:1397] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:57:36.392961 2076 ts_tablet_manager.cc:1397] T 3e995d5022594fc6b97b57fa8830d6fa P dc56d3b9199647d39d0352a17bdc1861: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 22:57:36.394017 2078 tablet_bootstrap.cc:492] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe: No bootstrap required, opened a new log
I20260928 22:57:36.394091 2078 ts_tablet_manager.cc:1397] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 22:57:36.394901 2077 raft_consensus.cc:348] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc56d3b9199647d39d0352a17bdc1861" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44451 } } peers { permanent_uuid: "d8d6cde850854019855d524224f1fdbe" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43691 } } peers { permanent_uuid: "49fe880a7cc246e1bb89926ccbff07cc" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40519 } }
I20260928 22:57:36.395061 2077 raft_consensus.cc:374] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:36.395104 2077 raft_consensus.cc:729] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49fe880a7cc246e1bb89926ccbff07cc, State: Initialized, Role: FOLLOWER
I20260928 22:57:36.395233 2077 consensus_queue.cc:260] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc56d3b9199647d39d0352a17bdc1861" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44451 } } peers { permanent_uuid: "d8d6cde850854019855d524224f1fdbe" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43691 } } peers { permanent_uuid: "49fe880a7cc246e1bb89926ccbff07cc" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40519 } }
I20260928 22:57:36.395521 2077 ts_tablet_manager.cc:1428] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:57:36.395668 2062 heartbeater.cc:503] Master 127.14.14.190:36579 was elected leader, sending a full tablet report...
I20260928 22:57:36.396201 2076 raft_consensus.cc:348] T 3e995d5022594fc6b97b57fa8830d6fa P dc56d3b9199647d39d0352a17bdc1861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc56d3b9199647d39d0352a17bdc1861" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44451 } } peers { permanent_uuid: "d8d6cde850854019855d524224f1fdbe" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43691 } } peers { permanent_uuid: "49fe880a7cc246e1bb89926ccbff07cc" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40519 } }
I20260928 22:57:36.396373 2076 raft_consensus.cc:374] T 3e995d5022594fc6b97b57fa8830d6fa P dc56d3b9199647d39d0352a17bdc1861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:36.396418 2076 raft_consensus.cc:729] T 3e995d5022594fc6b97b57fa8830d6fa P dc56d3b9199647d39d0352a17bdc1861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc56d3b9199647d39d0352a17bdc1861, State: Initialized, Role: FOLLOWER
I20260928 22:57:36.396641 2076 consensus_queue.cc:260] T 3e995d5022594fc6b97b57fa8830d6fa P dc56d3b9199647d39d0352a17bdc1861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc56d3b9199647d39d0352a17bdc1861" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44451 } } peers { permanent_uuid: "d8d6cde850854019855d524224f1fdbe" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43691 } } peers { permanent_uuid: "49fe880a7cc246e1bb89926ccbff07cc" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40519 } }
I20260928 22:57:36.396922 2076 ts_tablet_manager.cc:1428] T 3e995d5022594fc6b97b57fa8830d6fa P dc56d3b9199647d39d0352a17bdc1861: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 22:57:36.396914 2078 raft_consensus.cc:348] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc56d3b9199647d39d0352a17bdc1861" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44451 } } peers { permanent_uuid: "d8d6cde850854019855d524224f1fdbe" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43691 } } peers { permanent_uuid: "49fe880a7cc246e1bb89926ccbff07cc" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40519 } }
I20260928 22:57:36.397075 2078 raft_consensus.cc:374] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:36.397125 2078 raft_consensus.cc:729] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d6cde850854019855d524224f1fdbe, State: Initialized, Role: FOLLOWER
I20260928 22:57:36.397135 1800 heartbeater.cc:503] Master 127.14.14.190:36579 was elected leader, sending a full tablet report...
I20260928 22:57:36.397250 2078 consensus_queue.cc:260] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc56d3b9199647d39d0352a17bdc1861" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44451 } } peers { permanent_uuid: "d8d6cde850854019855d524224f1fdbe" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43691 } } peers { permanent_uuid: "49fe880a7cc246e1bb89926ccbff07cc" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40519 } }
I20260928 22:57:36.397567 2078 ts_tablet_manager.cc:1428] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 22:57:36.397670 1931 heartbeater.cc:503] Master 127.14.14.190:36579 was elected leader, sending a full tablet report...
W20260928 22:57:36.465478 1932 tablet.cc:2369] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:57:36.482123 2084 raft_consensus.cc:482] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:57:36.482254 2084 raft_consensus.cc:504] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc56d3b9199647d39d0352a17bdc1861" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44451 } } peers { permanent_uuid: "d8d6cde850854019855d524224f1fdbe" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43691 } } peers { permanent_uuid: "49fe880a7cc246e1bb89926ccbff07cc" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40519 } }
I20260928 22:57:36.482626 2084 leader_election.cc:288] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc56d3b9199647d39d0352a17bdc1861 (127.14.14.129:44451), 49fe880a7cc246e1bb89926ccbff07cc (127.14.14.131:40519)
I20260928 22:57:36.486649 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.482912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:48435 (local address 127.14.14.129:44451)
0928 22:57:36.482972 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:36.482974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:36.483010 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:57:36.483048 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:36.483049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:36.483058 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:36.483100 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:36.483298 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.484061 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.485629 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.486205 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.486279 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.486287 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:36.486531 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 22:57:36.486573 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:57:36.486656 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.482931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.14.130:42091 (local address 127.14.14.131:40519)
0928 22:57:36.482984 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:36.482986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:36.483182 (+ 196us) server_negotiation.cc:407] Connection header received
0928 22:57:36.483219 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:36.483220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:36.483227 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:36.483298 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:57:36.483427 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.484122 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.485644 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.486199 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.486279 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.486287 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:36.486531 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 22:57:36.486573 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:57:36.486728 1933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.482818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:44451 (local address 127.14.14.130:48435)
0928 22:57:36.482873 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:36.482885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:36.482936 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:36.483132 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:36.483132 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:36.483134 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:36.483213 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.483215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.484175 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.484176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.485521 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.485524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.486326 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.486327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.486423 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.486430 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 22:57:36.486508 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:57:36.486568 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":45}
I20260928 22:57:36.486735 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.482831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.131:40519 (local address 127.14.14.130:42091)
0928 22:57:36.483081 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:36.483112 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 22:57:36.483165 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:57:36.483311 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:57:36.483312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:57:36.483313 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:57:36.483366 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.483367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.484193 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.484194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.485522 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:57:36.485524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.486326 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.486327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:57:36.486423 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.486431 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 22:57:36.486504 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:57:36.486564 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":157,"thread_start_us":111,"threads_started":1}
I20260928 22:57:36.487154 1754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e995d5022594fc6b97b57fa8830d6fa" candidate_uuid: "d8d6cde850854019855d524224f1fdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc56d3b9199647d39d0352a17bdc1861" is_pre_election: true
I20260928 22:57:36.487236 2016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e995d5022594fc6b97b57fa8830d6fa" candidate_uuid: "d8d6cde850854019855d524224f1fdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fe880a7cc246e1bb89926ccbff07cc" is_pre_election: true
I20260928 22:57:36.487340 1754 raft_consensus.cc:2454] T 3e995d5022594fc6b97b57fa8830d6fa P dc56d3b9199647d39d0352a17bdc1861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d6cde850854019855d524224f1fdbe in term 0.
I20260928 22:57:36.487406 2016 raft_consensus.cc:2454] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d6cde850854019855d524224f1fdbe in term 0.
I20260928 22:57:36.487661 1818 leader_election.cc:302] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 49fe880a7cc246e1bb89926ccbff07cc, d8d6cde850854019855d524224f1fdbe, dc56d3b9199647d39d0352a17bdc1861; no voters:
I20260928 22:57:36.487807 2084 raft_consensus.cc:2781] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:57:36.487865 2084 raft_consensus.cc:482] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:57:36.487896 2084 raft_consensus.cc:3037] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:36.488559 2084 raft_consensus.cc:504] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc56d3b9199647d39d0352a17bdc1861" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44451 } } peers { permanent_uuid: "d8d6cde850854019855d524224f1fdbe" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43691 } } peers { permanent_uuid: "49fe880a7cc246e1bb89926ccbff07cc" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40519 } }
I20260928 22:57:36.488718 2084 leader_election.cc:288] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [CANDIDATE]: Term 1 election: Requested vote from peers dc56d3b9199647d39d0352a17bdc1861 (127.14.14.129:44451), 49fe880a7cc246e1bb89926ccbff07cc (127.14.14.131:40519)
I20260928 22:57:36.488850 1754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e995d5022594fc6b97b57fa8830d6fa" candidate_uuid: "d8d6cde850854019855d524224f1fdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc56d3b9199647d39d0352a17bdc1861"
I20260928 22:57:36.488933 1754 raft_consensus.cc:3037] T 3e995d5022594fc6b97b57fa8830d6fa P dc56d3b9199647d39d0352a17bdc1861 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:36.488937 2016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e995d5022594fc6b97b57fa8830d6fa" candidate_uuid: "d8d6cde850854019855d524224f1fdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fe880a7cc246e1bb89926ccbff07cc"
I20260928 22:57:36.489069 2016 raft_consensus.cc:3037] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:36.489714 1754 raft_consensus.cc:2454] T 3e995d5022594fc6b97b57fa8830d6fa P dc56d3b9199647d39d0352a17bdc1861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d6cde850854019855d524224f1fdbe in term 1.
I20260928 22:57:36.489876 1818 leader_election.cc:302] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8d6cde850854019855d524224f1fdbe, dc56d3b9199647d39d0352a17bdc1861; no voters:
I20260928 22:57:36.489939 2016 raft_consensus.cc:2454] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d6cde850854019855d524224f1fdbe in term 1.
I20260928 22:57:36.489979 2084 raft_consensus.cc:2781] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:36.490149 2084 raft_consensus.cc:686] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [term 1 LEADER]: Becoming Leader. State: Replica: d8d6cde850854019855d524224f1fdbe, State: Running, Role: LEADER
I20260928 22:57:36.490272 2084 consensus_queue.cc:237] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc56d3b9199647d39d0352a17bdc1861" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44451 } } peers { permanent_uuid: "d8d6cde850854019855d524224f1fdbe" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43691 } } peers { permanent_uuid: "49fe880a7cc246e1bb89926ccbff07cc" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40519 } }
I20260928 22:57:36.491200 1611 catalog_manager.cc:5495] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe reported cstate change: term changed from 0 to 1, leader changed from <none> to d8d6cde850854019855d524224f1fdbe (127.14.14.130). New cstate: current_term: 1 leader_uuid: "d8d6cde850854019855d524224f1fdbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc56d3b9199647d39d0352a17bdc1861" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d6cde850854019855d524224f1fdbe" member_type: VOTER last_known_addr { host: "127.14.14.130" port: 43691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49fe880a7cc246e1bb89926ccbff07cc" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40519 } health_report { overall_health: UNKNOWN } } }
I20260928 22:57:36.500509 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.498070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36004 (local address 127.14.14.130:43691)
0928 22:57:36.498129 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:36.498131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:36.498140 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:57:36.498175 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:36.498176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:36.498202 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:36.498281 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:57:36.498411 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.498880 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.499695 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:36.499891 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:36.499974 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:36.500153 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:57:36.500255 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:57:36.500352 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:36.500372 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 22:57:36.500392 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:57:36.502355 1754 raft_consensus.cc:1261] T 3e995d5022594fc6b97b57fa8830d6fa P dc56d3b9199647d39d0352a17bdc1861 [term 1 FOLLOWER]: Refusing update from remote peer d8d6cde850854019855d524224f1fdbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:57:36.502379 2016 raft_consensus.cc:1261] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc [term 1 FOLLOWER]: Refusing update from remote peer d8d6cde850854019855d524224f1fdbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:57:36.502737 2084 consensus_queue.cc:1035] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "49fe880a7cc246e1bb89926ccbff07cc" member_type: VOTER last_known_addr { host: "127.14.14.131" port: 40519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:57:36.502882 2086 consensus_queue.cc:1035] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc56d3b9199647d39d0352a17bdc1861" member_type: VOTER last_known_addr { host: "127.14.14.129" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:57:36.505038 2089 mvcc.cc:205] Tried to move back new op lower bound from 7334446106630758400 to 7334446106584920064. Current Snapshot: MvccSnapshot[applied={T|T < 7334446106630758400}]
I20260928 22:57:36.505262 2091 mvcc.cc:205] Tried to move back new op lower bound from 7334446106630758400 to 7334446106584920064. Current Snapshot: MvccSnapshot[applied={T|T < 7334446106630758400}]
I20260928 22:57:36.505463 2090 mvcc.cc:205] Tried to move back new op lower bound from 7334446106630758400 to 7334446106584920064. Current Snapshot: MvccSnapshot[applied={T|T < 7334446106630758400}]
I20260928 22:57:36.507635 1612 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:57:36.507730 1612 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:57:36.508464 1612 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8815a8e2560644a9979026da3b64f3ee: Sending DeleteTablet for 3 replicas of tablet 3e995d5022594fc6b97b57fa8830d6fa
I20260928 22:57:36.508807 1734 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e995d5022594fc6b97b57fa8830d6fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:36 UTC) from {username='slave'} at 127.0.0.1:44996
I20260928 22:57:36.508842 1865 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e995d5022594fc6b97b57fa8830d6fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:36 UTC) from {username='slave'} at 127.0.0.1:35998
I20260928 22:57:36.508898 1996 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e995d5022594fc6b97b57fa8830d6fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:57:36 UTC) from {username='slave'} at 127.0.0.1:52002
I20260928 22:57:36.509061 2100 tablet_replica.cc:330] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe: stopping tablet replica
I20260928 22:57:36.509124 2101 tablet_replica.cc:330] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc: stopping tablet replica
I20260928 22:57:36.509166 2100 raft_consensus.cc:2229] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:57:36.509249 2101 raft_consensus.cc:2229] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:57:36.509296 2100 raft_consensus.cc:2258] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:57:36.509342 2101 raft_consensus.cc:2258] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:57:36.509574 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 1672
I20260928 22:57:36.510226 2100 ts_tablet_manager.cc:1905] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:57:36.511061 2101 ts_tablet_manager.cc:1905] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:57:36.512331 2100 ts_tablet_manager.cc:1918] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:57:36.512394 2100 log.cc:1196] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-1/wal/wals/3e995d5022594fc6b97b57fa8830d6fa
I20260928 22:57:36.512662 2100 ts_tablet_manager.cc:1939] T 3e995d5022594fc6b97b57fa8830d6fa P d8d6cde850854019855d524224f1fdbe: Deleting consensus metadata
I20260928 22:57:36.512672 2101 ts_tablet_manager.cc:1918] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:57:36.512746 2101 log.cc:1196] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc: Deleting WAL directory at /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790636016256058-14394-0/minicluster-data/ts-2/wal/wals/3e995d5022594fc6b97b57fa8830d6fa
I20260928 22:57:36.513036 1596 catalog_manager.cc:4841] TS d8d6cde850854019855d524224f1fdbe (127.14.14.130:43691): tablet 3e995d5022594fc6b97b57fa8830d6fa (table test-table [id=8fd55a5195544d37af17880971053475]) successfully deleted
I20260928 22:57:36.513103 2101 ts_tablet_manager.cc:1939] T 3e995d5022594fc6b97b57fa8830d6fa P 49fe880a7cc246e1bb89926ccbff07cc: Deleting consensus metadata
I20260928 22:57:36.513494 1598 catalog_manager.cc:4841] TS 49fe880a7cc246e1bb89926ccbff07cc (127.14.14.131:40519): tablet 3e995d5022594fc6b97b57fa8830d6fa (table test-table [id=8fd55a5195544d37af17880971053475]) successfully deleted
W20260928 22:57:36.516326 1596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.14.129:44451 (error 108)
I20260928 22:57:36.516853 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:36.516708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.129:44451 (local address 127.0.0.1:45006)
0928 22:57:36.516771 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 22:57:36.516815 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.129:44451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 22:57:36.516884 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 1803
W20260928 22:57:36.518510 1596 catalog_manager.cc:4584] TS dc56d3b9199647d39d0352a17bdc1861 (127.14.14.129:44451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e995d5022594fc6b97b57fa8830d6fa: Network error: Client connection negotiation failed: client connection to 127.14.14.129:44451: connect: Connection refused (error 111)
I20260928 22:57:36.521982 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 1934
I20260928 22:57:36.526934 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 1581
2026-09-28T22:57:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (796 ms)
[----------] 6 tests from EncryptionPolicyTest (8632 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:36 dist-test-slave-kzfr krb5kdc[2105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:36 dist-test-slave-kzfr krb5kdc[2105](info): set up 2 sockets
Sep 28 22:57:36 dist-test-slave-kzfr krb5kdc[2105](info): commencing operation
krb5kdc: starting...
W20260928 22:57:37.593971 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:37 dist-test-slave-kzfr krb5kdc[2105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:57:37.716775 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35373
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:37.796005 2118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:37.796270 2118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:37.796309 2118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:37.797479 2118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:37.797564 2118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:37.797587 2118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:37.797605 2118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:37.797668 2118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:37.799185 2118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35373
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:37.799474 2118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:37.799715 2118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:37.806325 2126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:37.806331 2124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:37.806443 2123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:37.806756 2118 server_base.cc:1034] running on GCE node
I20260928 22:57:37.807032 2118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:37.807397 2118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:37.808589 2118 hybrid_clock.cc:648] HybridClock initialized: now 1790636257808577 us; error 35 us; skew 500 ppm
Sep 28 22:57:37 dist-test-slave-kzfr krb5kdc[2105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:37.814525 2118 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:57:37.815024 2118 webserver.cc:460] Webserver started at http://127.0.0.1:37361/ using document root <none> and password file <none>
I20260928 22:57:37.815294 2118 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:37.815366 2118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:37.815516 2118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:37.816597 2118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "7033097c413e4040bf7ba5596a8765a4"
format_stamp: "Formatted at 2026-09-28 22:57:37 on dist-test-slave-kzfr"
server_key: "2e318fe515b7bd4357c303bfedfc05dc"
server_key_iv: "35f666fec75bfc93a8e90b2d6f2d7d71"
server_key_version: "encryptionkey@0"
I20260928 22:57:37.816957 2118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "7033097c413e4040bf7ba5596a8765a4"
format_stamp: "Formatted at 2026-09-28 22:57:37 on dist-test-slave-kzfr"
server_key: "2e318fe515b7bd4357c303bfedfc05dc"
server_key_iv: "35f666fec75bfc93a8e90b2d6f2d7d71"
server_key_version: "encryptionkey@0"
I20260928 22:57:37.818491 2118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:57:37.819283 2133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:37.819479 2118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:37.819585 2118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "7033097c413e4040bf7ba5596a8765a4"
format_stamp: "Formatted at 2026-09-28 22:57:37 on dist-test-slave-kzfr"
server_key: "2e318fe515b7bd4357c303bfedfc05dc"
server_key_iv: "35f666fec75bfc93a8e90b2d6f2d7d71"
server_key_version: "encryptionkey@0"
I20260928 22:57:37.819660 2118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:37.831936 2118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:37.834270 2118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:37.834438 2118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:37.838711 2118 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35373
I20260928 22:57:37.838744 2185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35373 every 8 connection(s)
I20260928 22:57:37.839102 2118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:37.839979 2186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:37.841285 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 2118
I20260928 22:57:37.841408 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:37.841642 14394 external_mini_cluster.cc:1428] Setting key 041ba5cf3f9d97697de92995c7d62ff6
I20260928 22:57:37.842742 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4: Bootstrap starting.
I20260928 22:57:37.843477 2186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:37.843806 2186 log.cc:824] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:37.844537 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4: No bootstrap required, opened a new log
I20260928 22:57:37.846966 2186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7033097c413e4040bf7ba5596a8765a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35373 } }
I20260928 22:57:37.847158 2186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:37.847203 2186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7033097c413e4040bf7ba5596a8765a4, State: Initialized, Role: FOLLOWER
I20260928 22:57:37.847370 2186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7033097c413e4040bf7ba5596a8765a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35373 } }
I20260928 22:57:37.847463 2186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:37.847500 2186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:37.847548 2186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:37.848345 2186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7033097c413e4040bf7ba5596a8765a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35373 } }
I20260928 22:57:37.848434 2186 leader_election.cc:302] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7033097c413e4040bf7ba5596a8765a4; no voters:
I20260928 22:57:37.848613 2186 leader_election.cc:288] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:37.848670 2191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:37.848820 2191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [term 1 LEADER]: Becoming Leader. State: Replica: 7033097c413e4040bf7ba5596a8765a4, State: Running, Role: LEADER
I20260928 22:57:37.848878 2186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:37.848910 2191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7033097c413e4040bf7ba5596a8765a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35373 } }
Sep 28 22:57:37 dist-test-slave-kzfr krb5kdc[2105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 22:57:37.849254 2191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7033097c413e4040bf7ba5596a8765a4. Latest consensus state: current_term: 1 leader_uuid: "7033097c413e4040bf7ba5596a8765a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7033097c413e4040bf7ba5596a8765a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35373 } } }
I20260928 22:57:37.849261 2192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7033097c413e4040bf7ba5596a8765a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7033097c413e4040bf7ba5596a8765a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35373 } } }
I20260928 22:57:37.849339 2191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:37.849360 2192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4 [sys.catalog]: This master's current role is: LEADER
W20260928 22:57:37.851014 2206 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:57:37.851083 2206 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:57:37.851145 2207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:37.851337 2207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:37.852926 2207 catalog_manager.cc:1347] Generated new cluster ID: c9040d1cdf7143f595915df285d6fc41
I20260928 22:57:37.852978 2207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:37.853752 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:37.842185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38505 (local address 127.0.0.1:35373)
0928 22:57:37.842361 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:37.842370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:37.842400 (+ 30us) server_negotiation.cc:407] Connection header received
0928 22:57:37.842824 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:37.842833 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:37.842897 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:37.843011 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:37.843907 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:37.844998 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:37.846103 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:37.846521 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:37.846604 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:37.849606 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:37.849611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:37.849611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:37.849624 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:37.851830 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:37.852341 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:37.852345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:37.852345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:37.852463 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:37.852741 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:37.852742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:37.852743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:37.853103 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:37.853289 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:37.853417 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 22:57:37.853517 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260928 22:57:37.854697 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:37.863795 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:37.856026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53147 (local address 127.0.0.1:35373)
0928 22:57:37.856062 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:37.856063 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:37.857829 (+ 1766us) server_negotiation.cc:407] Connection header received
0928 22:57:37.857893 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:37.857894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:37.857905 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:37.857959 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:37.858485 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:37.858984 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:37.859706 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:37.859928 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:37.860002 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:37.861193 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:37.861194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:37.861195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:37.861202 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:37.862633 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:37.862997 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:37.862999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:37.862999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:37.863061 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:37.863285 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:37.863287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:37.863287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:37.863402 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:37.863515 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:37.863629 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 22:57:37.863691 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:57:37.874588 2207 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:37.875133 2207 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:37.881664 2207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7033097c413e4040bf7ba5596a8765a4: Generated new TSK 0
I20260928 22:57:37.881877 2207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:37.888492 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 2118
W20260928 22:57:37.892967 2103 connection.cc:535] client connection to 127.0.0.1:35373 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35373): Connection reset by peer (error 104)
2026-09-28T22:57:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:37 dist-test-slave-kzfr krb5kdc[2217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:37 dist-test-slave-kzfr krb5kdc[2217](info): set up 2 sockets
Sep 28 22:57:37 dist-test-slave-kzfr krb5kdc[2217](info): commencing operation
krb5kdc: starting...
W20260928 22:57:38.941901 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:39 dist-test-slave-kzfr krb5kdc[2217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:57:39.064810 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:39.146440 2230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:39.146651 2230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:39.146673 2230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:39.147825 2230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:39.147867 2230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:39.147882 2230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:39.147895 2230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:39.147940 2230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:39.149395 2230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39349
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:39.149678 2230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:39.149904 2230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:39.154781 2235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:39.154781 2238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:39.154879 2230 server_base.cc:1034] running on GCE node
W20260928 22:57:39.154831 2236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:39.155246 2230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:39.155539 2230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:39.156706 2230 hybrid_clock.cc:648] HybridClock initialized: now 1790636259156695 us; error 30 us; skew 500 ppm
Sep 28 22:57:39 dist-test-slave-kzfr krb5kdc[2217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:39.159176 2230 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:57:39.159693 2230 webserver.cc:460] Webserver started at http://127.0.0.1:43579/ using document root <none> and password file <none>
I20260928 22:57:39.159934 2230 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:39.160022 2230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:39.160162 2230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:39.161762 2230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "2c458e3669ca4a90b2e9995552304397"
format_stamp: "Formatted at 2026-09-28 22:57:39 on dist-test-slave-kzfr"
server_key: "a2d295a7de4a065d4c1095ea88daa6ed"
server_key_iv: "4e2602a2e004c19f84e90fc6479eebad"
server_key_version: "encryptionkey@0"
I20260928 22:57:39.162180 2230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "2c458e3669ca4a90b2e9995552304397"
format_stamp: "Formatted at 2026-09-28 22:57:39 on dist-test-slave-kzfr"
server_key: "a2d295a7de4a065d4c1095ea88daa6ed"
server_key_iv: "4e2602a2e004c19f84e90fc6479eebad"
server_key_version: "encryptionkey@0"
I20260928 22:57:39.165930 2230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 22:57:39.167840 2245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:39.167991 2230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 22:57:39.168066 2230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "2c458e3669ca4a90b2e9995552304397"
format_stamp: "Formatted at 2026-09-28 22:57:39 on dist-test-slave-kzfr"
server_key: "a2d295a7de4a065d4c1095ea88daa6ed"
server_key_iv: "4e2602a2e004c19f84e90fc6479eebad"
server_key_version: "encryptionkey@0"
I20260928 22:57:39.168143 2230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:39.181175 2230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:39.183106 2230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:39.183218 2230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:39.187018 2230 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39349
I20260928 22:57:39.187081 2297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39349 every 8 connection(s)
I20260928 22:57:39.187445 2230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:39.188333 2298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:39.189720 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 2230
I20260928 22:57:39.189841 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:39.190044 14394 external_mini_cluster.cc:1428] Setting key 88f8bf8df4602c77663abfc0a2f08cc7
I20260928 22:57:39.191035 2298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397: Bootstrap starting.
I20260928 22:57:39.191593 2298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:39.191886 2298 log.cc:824] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:39.192566 2298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397: No bootstrap required, opened a new log
I20260928 22:57:39.194403 2298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c458e3669ca4a90b2e9995552304397" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39349 } }
I20260928 22:57:39.194576 2298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:39.194617 2298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c458e3669ca4a90b2e9995552304397, State: Initialized, Role: FOLLOWER
I20260928 22:57:39.194779 2298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c458e3669ca4a90b2e9995552304397" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39349 } }
I20260928 22:57:39.194881 2298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:39.194931 2298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:39.194980 2298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:39.195500 2301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:39.190547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54313 (local address 127.0.0.1:39349)
0928 22:57:39.190760 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:39.190777 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:39.190802 (+ 25us) server_negotiation.cc:407] Connection header received
0928 22:57:39.191282 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:39.191289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:39.191337 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:39.191486 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:39.191747 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:39.192744 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:39.193554 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:39.193999 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:39.194122 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:39.194837 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:39.194841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:39.194842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:39.195067 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:39.195128 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:39.195139 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:39.195229 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:39.195310 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 22:57:39.195340 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":102,"threads_started":1}
I20260928 22:57:39.195640 2298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c458e3669ca4a90b2e9995552304397" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39349 } }
I20260928 22:57:39.195746 2298 leader_election.cc:302] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c458e3669ca4a90b2e9995552304397; no voters:
I20260928 22:57:39.195910 2298 leader_election.cc:288] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:39.195972 2303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:39.196151 2303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [term 1 LEADER]: Becoming Leader. State: Replica: 2c458e3669ca4a90b2e9995552304397, State: Running, Role: LEADER
I20260928 22:57:39.196241 2303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c458e3669ca4a90b2e9995552304397" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39349 } }
I20260928 22:57:39.196300 2298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:39.196547 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 22:57:39.198019 2318 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:57:39.198079 2318 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:57:39.198159 2304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c458e3669ca4a90b2e9995552304397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c458e3669ca4a90b2e9995552304397" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39349 } } }
I20260928 22:57:39.198230 2304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:39.198423 2324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:39.198621 2324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:39.198405 2305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c458e3669ca4a90b2e9995552304397. Latest consensus state: current_term: 1 leader_uuid: "2c458e3669ca4a90b2e9995552304397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c458e3669ca4a90b2e9995552304397" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39349 } } }
I20260928 22:57:39.199496 2305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:39.200387 2324 catalog_manager.cc:1347] Generated new cluster ID: 4fe4e9ec4a3a4003a98374b30547efb2
I20260928 22:57:39.200440 2324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:39.202324 2301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:39.198746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44643 (local address 127.0.0.1:39349)
0928 22:57:39.198785 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:39.198787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:39.198868 (+ 81us) server_negotiation.cc:407] Connection header received
0928 22:57:39.198994 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:39.198995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:39.199006 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:39.199069 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:39.199231 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:39.199914 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:39.200905 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:39.201224 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:39.201303 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:39.202036 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:39.202038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:39.202038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:39.202043 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:39.202059 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:39.202063 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:39.202136 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:39.202233 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 22:57:39.202245 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:57:39.212110 2324 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:39.212862 2324 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:39.224471 2324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c458e3669ca4a90b2e9995552304397: Generated new TSK 0
I20260928 22:57:39.224725 2324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:39.231657 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 2230
W20260928 22:57:39.236017 2215 connection.cc:535] client connection to 127.0.0.1:39349 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39349): Connection reset by peer (error 104)
2026-09-28T22:57:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:39 dist-test-slave-kzfr krb5kdc[2329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:39 dist-test-slave-kzfr krb5kdc[2329](info): set up 2 sockets
Sep 28 22:57:39 dist-test-slave-kzfr krb5kdc[2329](info): commencing operation
krb5kdc: starting...
W20260928 22:57:40.262323 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:40 dist-test-slave-kzfr krb5kdc[2329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:57:40.381309 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38219
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:40.461452 2342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:40.461727 2342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:40.461766 2342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:40.462895 2342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:40.462971 2342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:40.462996 2342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:40.463017 2342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:40.463048 2342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:40.464524 2342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38219
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:40.464815 2342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:40.465049 2342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:40.469894 2348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:40.469887 2350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:40.469892 2347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:40.470098 2342 server_base.cc:1034] running on GCE node
I20260928 22:57:40.470326 2342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:40.470606 2342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:40.471788 2342 hybrid_clock.cc:648] HybridClock initialized: now 1790636260471776 us; error 51 us; skew 500 ppm
Sep 28 22:57:40 dist-test-slave-kzfr krb5kdc[2329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:40.474220 2342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:57:40.474706 2342 webserver.cc:460] Webserver started at http://127.0.0.1:44247/ using document root <none> and password file <none>
I20260928 22:57:40.474936 2342 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:40.475004 2342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:40.475164 2342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:40.476248 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "c7c24a2ec0e440c4a167b8997ae551ff"
format_stamp: "Formatted at 2026-09-28 22:57:40 on dist-test-slave-kzfr"
server_key: "87622ed201f29483b0ba7c9ba8c4734d"
server_key_iv: "b01a9f8ed37887b4df4b7aaa3c5e47a9"
server_key_version: "encryptionkey@0"
I20260928 22:57:40.476621 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "c7c24a2ec0e440c4a167b8997ae551ff"
format_stamp: "Formatted at 2026-09-28 22:57:40 on dist-test-slave-kzfr"
server_key: "87622ed201f29483b0ba7c9ba8c4734d"
server_key_iv: "b01a9f8ed37887b4df4b7aaa3c5e47a9"
server_key_version: "encryptionkey@0"
I20260928 22:57:40.478024 2342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:57:40.478765 2357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:40.478937 2342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:40.479046 2342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "c7c24a2ec0e440c4a167b8997ae551ff"
format_stamp: "Formatted at 2026-09-28 22:57:40 on dist-test-slave-kzfr"
server_key: "87622ed201f29483b0ba7c9ba8c4734d"
server_key_iv: "b01a9f8ed37887b4df4b7aaa3c5e47a9"
server_key_version: "encryptionkey@0"
I20260928 22:57:40.479125 2342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:40.482502 2342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:40.484391 2342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:40.484550 2342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:40.488699 2342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38219
I20260928 22:57:40.488729 2409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38219 every 8 connection(s)
I20260928 22:57:40.489058 2342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:40.489970 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:40.492167 2410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff: Bootstrap starting.
I20260928 22:57:40.492777 2410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:40.493062 2410 log.cc:824] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:40.493832 2410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff: No bootstrap required, opened a new log
I20260928 22:57:40.495391 2410 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c24a2ec0e440c4a167b8997ae551ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } }
I20260928 22:57:40.495527 2410 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:40.495575 2410 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c24a2ec0e440c4a167b8997ae551ff, State: Initialized, Role: FOLLOWER
I20260928 22:57:40.495770 2410 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c24a2ec0e440c4a167b8997ae551ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } }
I20260928 22:57:40.495889 2410 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:40.495935 2410 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:40.495985 2410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:40.496057 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 2342
I20260928 22:57:40.496209 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:40.496394 14394 external_mini_cluster.cc:1428] Setting key ad4804f82bd8bea99a9056b182ee5967
I20260928 22:57:40.496704 2410 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c24a2ec0e440c4a167b8997ae551ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } }
I20260928 22:57:40.496822 2410 leader_election.cc:302] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7c24a2ec0e440c4a167b8997ae551ff; no voters:
I20260928 22:57:40.496989 2410 leader_election.cc:288] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:40.497014 2414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:40.497206 2414 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [term 1 LEADER]: Becoming Leader. State: Replica: c7c24a2ec0e440c4a167b8997ae551ff, State: Running, Role: LEADER
I20260928 22:57:40.497284 2410 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:40.497334 2414 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c24a2ec0e440c4a167b8997ae551ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } }
I20260928 22:57:40.497802 2416 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7c24a2ec0e440c4a167b8997ae551ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c24a2ec0e440c4a167b8997ae551ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } } }
I20260928 22:57:40.497990 2416 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:40.498083 2414 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7c24a2ec0e440c4a167b8997ae551ff. Latest consensus state: current_term: 1 leader_uuid: "c7c24a2ec0e440c4a167b8997ae551ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c24a2ec0e440c4a167b8997ae551ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } } }
I20260928 22:57:40.498294 2414 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:40.498523 2425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:40.498873 2415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:40.496879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58593 (local address 127.0.0.1:38219)
0928 22:57:40.497161 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:40.497170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:40.497203 (+ 33us) server_negotiation.cc:407] Connection header received
0928 22:57:40.497312 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:40.497319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:40.497349 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:40.497505 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:40.498255 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:40.498263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:40.498263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:40.498272 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:40.498310 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:40.498326 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:40.498423 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:40.498650 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 22:57:40.498693 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":100,"threads_started":1}
I20260928 22:57:40.499375 2425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:40.499962 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:40.501061 2425 catalog_manager.cc:1347] Generated new cluster ID: de97e5b2dd9541c5a403ba27fa6eb7ce
I20260928 22:57:40.501134 2425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:40.503103 2415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:40.501968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36869 (local address 127.0.0.1:38219)
0928 22:57:40.502001 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:40.502003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:40.502117 (+ 114us) server_negotiation.cc:407] Connection header received
0928 22:57:40.502208 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:40.502208 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:40.502213 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:40.502239 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:40.502881 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:40.502882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:40.502882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:40.502886 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:40.502899 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:40.502903 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:40.502929 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:40.503022 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 22:57:40.503033 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 22:57:40.504941 2425 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:40.505102 2425 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:40.509351 2425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7c24a2ec0e440c4a167b8997ae551ff: Generated new TSK 0
I20260928 22:57:40.509528 2425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:40.514842 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 2342
2026-09-28T22:57:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:40 dist-test-slave-kzfr krb5kdc[2441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:40 dist-test-slave-kzfr krb5kdc[2441](info): set up 2 sockets
Sep 28 22:57:40 dist-test-slave-kzfr krb5kdc[2441](info): commencing operation
krb5kdc: starting...
W20260928 22:57:41.549902 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:41 dist-test-slave-kzfr krb5kdc[2441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:57:41.672452 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36095
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:41.753160 2454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:41.753420 2454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:41.753471 2454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:41.754712 2454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:41.754791 2454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:41.754815 2454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:41.754835 2454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:41.754865 2454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:41.756356 2454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:41.756682 2454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:41.756927 2454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:41.762013 2460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:41.762044 2462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:41.762044 2459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:41.762185 2454 server_base.cc:1034] running on GCE node
I20260928 22:57:41.762502 2454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:41.762813 2454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:41.764030 2454 hybrid_clock.cc:648] HybridClock initialized: now 1790636261764016 us; error 45 us; skew 500 ppm
Sep 28 22:57:41 dist-test-slave-kzfr krb5kdc[2441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:41.766685 2454 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:57:41.767148 2454 webserver.cc:460] Webserver started at http://127.0.0.1:46513/ using document root <none> and password file <none>
I20260928 22:57:41.767373 2454 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:41.767418 2454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:41.767544 2454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:41.768589 2454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "894c450550744212b5d7c25f8647e7ec"
format_stamp: "Formatted at 2026-09-28 22:57:41 on dist-test-slave-kzfr"
server_key: "74af3cd8e050365388c5e18e7860c938"
server_key_iv: "797239c7cd6571fdfdcc1ecd34070456"
server_key_version: "encryptionkey@0"
I20260928 22:57:41.768949 2454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "894c450550744212b5d7c25f8647e7ec"
format_stamp: "Formatted at 2026-09-28 22:57:41 on dist-test-slave-kzfr"
server_key: "74af3cd8e050365388c5e18e7860c938"
server_key_iv: "797239c7cd6571fdfdcc1ecd34070456"
server_key_version: "encryptionkey@0"
I20260928 22:57:41.770373 2454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:57:41.771133 2469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:41.771309 2454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:41.771411 2454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "894c450550744212b5d7c25f8647e7ec"
format_stamp: "Formatted at 2026-09-28 22:57:41 on dist-test-slave-kzfr"
server_key: "74af3cd8e050365388c5e18e7860c938"
server_key_iv: "797239c7cd6571fdfdcc1ecd34070456"
server_key_version: "encryptionkey@0"
I20260928 22:57:41.771492 2454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:41.779177 2454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:41.781457 2454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:41.781672 2454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:41.785872 2454 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36095
I20260928 22:57:41.786002 2521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36095 every 8 connection(s)
I20260928 22:57:41.786293 2454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:41.787037 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 2454
I20260928 22:57:41.787195 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:41.787238 2522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:41.787420 14394 external_mini_cluster.cc:1428] Setting key 5e8516f2ca7a1c79a2efcba4524ae312
I20260928 22:57:41.790035 2522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec: Bootstrap starting.
I20260928 22:57:41.790607 2522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:41.790885 2522 log.cc:824] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:41.791545 2522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec: No bootstrap required, opened a new log
I20260928 22:57:41.793311 2522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c450550744212b5d7c25f8647e7ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36095 } }
I20260928 22:57:41.793489 2522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:41.793546 2522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894c450550744212b5d7c25f8647e7ec, State: Initialized, Role: FOLLOWER
I20260928 22:57:41.793730 2522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c450550744212b5d7c25f8647e7ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36095 } }
I20260928 22:57:41.793826 2522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:41.793871 2522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 28 22:57:41 dist-test-slave-kzfr krb5kdc[2441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 22:57:41.793926 2522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:41.794713 2522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c450550744212b5d7c25f8647e7ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36095 } }
I20260928 22:57:41.794831 2522 leader_election.cc:302] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 894c450550744212b5d7c25f8647e7ec; no voters:
I20260928 22:57:41.795019 2522 leader_election.cc:288] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:41.795082 2527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:41.795279 2522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:41.795277 2527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [term 1 LEADER]: Becoming Leader. State: Replica: 894c450550744212b5d7c25f8647e7ec, State: Running, Role: LEADER
I20260928 22:57:41.795498 2527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c450550744212b5d7c25f8647e7ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36095 } }
I20260928 22:57:41.795874 2528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "894c450550744212b5d7c25f8647e7ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c450550744212b5d7c25f8647e7ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36095 } } }
I20260928 22:57:41.795889 2529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 894c450550744212b5d7c25f8647e7ec. Latest consensus state: current_term: 1 leader_uuid: "894c450550744212b5d7c25f8647e7ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c450550744212b5d7c25f8647e7ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36095 } } }
I20260928 22:57:41.796080 2528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:41.796088 2529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:41.796336 2536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:41.796926 2536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:41.798593 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:41.787941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34243 (local address 127.0.0.1:36095)
0928 22:57:41.788121 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:41.788130 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:41.788164 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:57:41.788598 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:41.788604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:41.788657 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:41.788806 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:41.789427 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:41.790470 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:41.791352 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:41.791817 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:41.791938 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:41.794545 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:41.794551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:41.794551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:41.794561 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:41.796811 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:41.797238 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:41.797241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:41.797242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:41.797346 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:41.797576 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:41.797577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:41.797577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:41.797921 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:41.798061 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:41.798251 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 22:57:41.798404 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260928 22:57:41.798715 2536 catalog_manager.cc:1347] Generated new cluster ID: 7b7cc626e2334a7ead9851e7fd989279
I20260928 22:57:41.798755 2536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:41.799546 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:41.817570 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:41.809934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42695 (local address 127.0.0.1:36095)
0928 22:57:41.809988 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:41.809991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:41.810156 (+ 165us) server_negotiation.cc:407] Connection header received
0928 22:57:41.810229 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:41.810230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:41.810240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:41.810302 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:41.810990 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:41.811745 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:41.812802 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:41.813125 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:41.813189 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:41.814715 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:41.814716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:41.814717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:41.814724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:41.816253 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:41.816808 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:41.816809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:41.816809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:41.816862 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:41.817064 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:41.817065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:41.817065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:41.817187 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:41.817292 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:41.817409 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 22:57:41.817475 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:57:41.823451 2536 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:41.824201 2536 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:41.826702 2536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 894c450550744212b5d7c25f8647e7ec: Generated new TSK 0
I20260928 22:57:41.826896 2536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:41.834692 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 2454
2026-09-28T22:57:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:41 dist-test-slave-kzfr krb5kdc[2553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:41 dist-test-slave-kzfr krb5kdc[2553](info): set up 2 sockets
Sep 28 22:57:41 dist-test-slave-kzfr krb5kdc[2553](info): commencing operation
krb5kdc: starting...
W20260928 22:57:42.929920 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:43 dist-test-slave-kzfr krb5kdc[2553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:57:43.052384 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:43.133167 2566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:43.133428 2566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:43.133466 2566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:43.134689 2566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:43.134768 2566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:43.134792 2566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:43.134819 2566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:43.134851 2566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:43.136307 2566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:43.136581 2566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:43.136807 2566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:57:43.141909 2566 server_base.cc:1034] running on GCE node
W20260928 22:57:43.141924 2572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:43.142037 2571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:43.142191 2574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:43.142421 2566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:43.142696 2566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:43.143863 2566 hybrid_clock.cc:648] HybridClock initialized: now 1790636263143849 us; error 25 us; skew 500 ppm
Sep 28 22:57:43 dist-test-slave-kzfr krb5kdc[2553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:43.149667 2566 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:57:43.150179 2566 webserver.cc:460] Webserver started at http://127.0.0.1:45757/ using document root <none> and password file <none>
I20260928 22:57:43.150365 2566 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:43.150405 2566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:43.150494 2566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:43.151494 2566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "68c4389aa10942ec95a3c73226def578"
format_stamp: "Formatted at 2026-09-28 22:57:43 on dist-test-slave-kzfr"
server_key: "5acd6e27eabffa441b0f5348983359d0"
server_key_iv: "91cd01c5875100bb4188455e7609c0df"
server_key_version: "encryptionkey@0"
I20260928 22:57:43.151881 2566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "68c4389aa10942ec95a3c73226def578"
format_stamp: "Formatted at 2026-09-28 22:57:43 on dist-test-slave-kzfr"
server_key: "5acd6e27eabffa441b0f5348983359d0"
server_key_iv: "91cd01c5875100bb4188455e7609c0df"
server_key_version: "encryptionkey@0"
I20260928 22:57:43.153214 2566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:57:43.154012 2581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:43.154217 2566 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:43.154322 2566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "68c4389aa10942ec95a3c73226def578"
format_stamp: "Formatted at 2026-09-28 22:57:43 on dist-test-slave-kzfr"
server_key: "5acd6e27eabffa441b0f5348983359d0"
server_key_iv: "91cd01c5875100bb4188455e7609c0df"
server_key_version: "encryptionkey@0"
I20260928 22:57:43.154395 2566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:43.168725 2566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:43.171135 2566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:43.171299 2566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:43.175336 2566 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44831
I20260928 22:57:43.175374 2633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44831 every 8 connection(s)
I20260928 22:57:43.175757 2566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:43.176676 2634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:43.177124 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 2566
I20260928 22:57:43.177266 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:43.177517 14394 external_mini_cluster.cc:1428] Setting key 70e7440dc095d06e31257962b21973fa
I20260928 22:57:43.180336 2634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578: Bootstrap starting.
I20260928 22:57:43.181125 2634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:43.181523 2634 log.cc:824] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:43.182380 2634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578: No bootstrap required, opened a new log
I20260928 22:57:43.184402 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:43.178213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41149 (local address 127.0.0.1:44831)
0928 22:57:43.178449 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:43.178458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:43.178687 (+ 229us) server_negotiation.cc:407] Connection header received
0928 22:57:43.179291 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:43.179300 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:43.179376 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:43.179542 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:43.179882 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:43.181298 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:43.182257 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:43.182849 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:43.182978 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:43.183669 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:43.183673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:43.183674 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:43.183885 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:43.183928 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:43.183954 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:43.184019 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:43.184112 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 22:57:43.184180 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":93,"threads_started":1}
I20260928 22:57:43.185122 2634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c4389aa10942ec95a3c73226def578" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260928 22:57:43.185319 2634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:43.185361 2634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c4389aa10942ec95a3c73226def578, State: Initialized, Role: FOLLOWER
I20260928 22:57:43.185491 2634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c4389aa10942ec95a3c73226def578" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260928 22:57:43.185599 2634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:43.185550 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:43.185645 2634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:43.185695 2634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:43.186259 2634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c4389aa10942ec95a3c73226def578" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260928 22:57:43.186381 2634 leader_election.cc:302] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68c4389aa10942ec95a3c73226def578; no voters:
I20260928 22:57:43.186578 2634 leader_election.cc:288] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:43.186625 2640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:43.186828 2634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:43.186866 2640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [term 1 LEADER]: Becoming Leader. State: Replica: 68c4389aa10942ec95a3c73226def578, State: Running, Role: LEADER
I20260928 22:57:43.187371 2640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c4389aa10942ec95a3c73226def578" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260928 22:57:43.188793 2643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68c4389aa10942ec95a3c73226def578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c4389aa10942ec95a3c73226def578" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } } }
I20260928 22:57:43.188800 2645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68c4389aa10942ec95a3c73226def578. Latest consensus state: current_term: 1 leader_uuid: "68c4389aa10942ec95a3c73226def578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c4389aa10942ec95a3c73226def578" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } } }
I20260928 22:57:43.188891 2643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:43.188906 2645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578 [sys.catalog]: This master's current role is: LEADER
W20260928 22:57:43.188911 2658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:57:43.188971 2658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:57:43.189090 2661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:43.189270 2661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:43.190680 2661 catalog_manager.cc:1347] Generated new cluster ID: 6d96686bb6b846e78ceff7b2ad1df0a8
I20260928 22:57:43.190729 2661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:43.191705 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:43.187127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41357 (local address 127.0.0.1:44831)
0928 22:57:43.187187 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:43.187189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:43.187383 (+ 194us) server_negotiation.cc:407] Connection header received
0928 22:57:43.187490 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:43.187492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:43.187503 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:43.187558 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:43.187761 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:43.188509 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:43.190127 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:43.190401 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:43.190443 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:43.191415 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:43.191416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:43.191417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:43.191422 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:43.191437 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:43.191443 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:43.191472 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:43.191588 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 22:57:43.191614 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:57:43.203543 2661 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:43.204382 2661 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:43.216215 2661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68c4389aa10942ec95a3c73226def578: Generated new TSK 0
I20260928 22:57:43.216461 2661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:43.228027 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 2566
2026-09-28T22:57:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:43 dist-test-slave-kzfr krb5kdc[2665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:43 dist-test-slave-kzfr krb5kdc[2665](info): set up 2 sockets
Sep 28 22:57:43 dist-test-slave-kzfr krb5kdc[2665](info): commencing operation
krb5kdc: starting...
W20260928 22:57:44.265964 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:44 dist-test-slave-kzfr krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:57:44.385324 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:44.466331 2678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:44.466526 2678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:44.466547 2678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:44.467713 2678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:44.467760 2678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:44.467775 2678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:44.467816 2678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:44.467847 2678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:44.469316 2678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:44.469574 2678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:44.469857 2678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:44.474718 2683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:44.474718 2684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:44.474758 2686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:44.474980 2678 server_base.cc:1034] running on GCE node
I20260928 22:57:44.475209 2678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:44.475451 2678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:44.476612 2678 hybrid_clock.cc:648] HybridClock initialized: now 1790636264476585 us; error 39 us; skew 500 ppm
Sep 28 22:57:44 dist-test-slave-kzfr krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:44.479038 2678 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:57:44.479503 2678 webserver.cc:460] Webserver started at http://127.0.0.1:33315/ using document root <none> and password file <none>
I20260928 22:57:44.479734 2678 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:44.479802 2678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:44.479955 2678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:44.481021 2678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "873525f258b1409f8ccd2dc465918840"
format_stamp: "Formatted at 2026-09-28 22:57:44 on dist-test-slave-kzfr"
server_key: "a34e7340484abd6f9731e9799dc643a7"
server_key_iv: "2812e54d84b750958402d009b72073a1"
server_key_version: "encryptionkey@0"
I20260928 22:57:44.481384 2678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "873525f258b1409f8ccd2dc465918840"
format_stamp: "Formatted at 2026-09-28 22:57:44 on dist-test-slave-kzfr"
server_key: "a34e7340484abd6f9731e9799dc643a7"
server_key_iv: "2812e54d84b750958402d009b72073a1"
server_key_version: "encryptionkey@0"
I20260928 22:57:44.482781 2678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:57:44.483610 2693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:44.483834 2678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:57:44.483932 2678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "873525f258b1409f8ccd2dc465918840"
format_stamp: "Formatted at 2026-09-28 22:57:44 on dist-test-slave-kzfr"
server_key: "a34e7340484abd6f9731e9799dc643a7"
server_key_iv: "2812e54d84b750958402d009b72073a1"
server_key_version: "encryptionkey@0"
I20260928 22:57:44.484015 2678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:44.487349 2678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:44.489117 2678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:44.489269 2678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:44.493232 2678 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42245
I20260928 22:57:44.493264 2745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42245 every 8 connection(s)
I20260928 22:57:44.493628 2678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:44.494832 2746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:44.497072 2746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840: Bootstrap starting.
I20260928 22:57:44.497714 2746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:44.497999 2746 log.cc:824] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:44.498705 2746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840: No bootstrap required, opened a new log
I20260928 22:57:44.500118 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 2678
I20260928 22:57:44.500262 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:44.500447 14394 external_mini_cluster.cc:1428] Setting key 8964596a62609745bd1bc353b7ec698d
I20260928 22:57:44.500690 2746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873525f258b1409f8ccd2dc465918840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42245 } }
I20260928 22:57:44.500878 2746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:44.500922 2746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873525f258b1409f8ccd2dc465918840, State: Initialized, Role: FOLLOWER
I20260928 22:57:44.501037 2746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873525f258b1409f8ccd2dc465918840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42245 } }
I20260928 22:57:44.501130 2746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:44.501179 2746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:44.501261 2746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:44.502501 2746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873525f258b1409f8ccd2dc465918840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42245 } }
I20260928 22:57:44.502595 2746 leader_election.cc:302] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 873525f258b1409f8ccd2dc465918840; no voters:
I20260928 22:57:44.502780 2746 leader_election.cc:288] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:44.502971 2752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:44.503060 2746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:44.503115 2752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [term 1 LEADER]: Becoming Leader. State: Replica: 873525f258b1409f8ccd2dc465918840, State: Running, Role: LEADER
I20260928 22:57:44.503202 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:44.501048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44705 (local address 127.0.0.1:42245)
0928 22:57:44.501267 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:44.501275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:44.501322 (+ 47us) server_negotiation.cc:407] Connection header received
0928 22:57:44.501531 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:44.501540 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:44.501568 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:44.501746 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:44.502420 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:44.502424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:44.502424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:44.502427 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:44.502543 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:44.502576 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:44.502658 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:44.502794 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 22:57:44.502881 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":109,"threads_started":1}
I20260928 22:57:44.503296 2752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873525f258b1409f8ccd2dc465918840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42245 } }
I20260928 22:57:44.503826 2752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 873525f258b1409f8ccd2dc465918840. Latest consensus state: current_term: 1 leader_uuid: "873525f258b1409f8ccd2dc465918840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873525f258b1409f8ccd2dc465918840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42245 } } }
I20260928 22:57:44.503943 2752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:44.503927 2753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "873525f258b1409f8ccd2dc465918840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873525f258b1409f8ccd2dc465918840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42245 } } }
I20260928 22:57:44.504086 2753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:44.504114 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:44.504628 2760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:44.505138 2760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:44.506799 2760 catalog_manager.cc:1347] Generated new cluster ID: 6f2f454d93ce422aaeab7f056091b251
I20260928 22:57:44.506856 2760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:44.507144 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:44.505688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37499 (local address 127.0.0.1:42245)
0928 22:57:44.505722 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:44.505724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:44.506111 (+ 387us) server_negotiation.cc:407] Connection header received
0928 22:57:44.506160 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:44.506161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:44.506166 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:44.506230 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:44.506912 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:44.506913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:44.506914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:44.506918 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:44.506931 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:44.506935 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:44.506965 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:44.507073 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 22:57:44.507087 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:57:44.520421 2760 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:44.520597 2760 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:44.525665 2760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 873525f258b1409f8ccd2dc465918840: Generated new TSK 0
I20260928 22:57:44.525876 2760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:44.535343 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 2678
2026-09-28T22:57:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:44 dist-test-slave-kzfr krb5kdc[2777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:44 dist-test-slave-kzfr krb5kdc[2777](info): set up 2 sockets
Sep 28 22:57:44 dist-test-slave-kzfr krb5kdc[2777](info): commencing operation
krb5kdc: starting...
W20260928 22:57:45.609970 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:45 dist-test-slave-kzfr krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:57:45.728953 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:45.813144 2790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:45.813357 2790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:45.813381 2790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:45.814642 2790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:45.814698 2790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:45.814714 2790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:45.814765 2790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:45.814786 2790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:45.816289 2790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:45.816572 2790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:45.816813 2790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:45.821796 2795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:45.821818 2796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:45.821849 2790 server_base.cc:1034] running on GCE node
W20260928 22:57:45.821820 2798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:45.822227 2790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:45.822516 2790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:45.823666 2790 hybrid_clock.cc:648] HybridClock initialized: now 1790636265823651 us; error 27 us; skew 500 ppm
Sep 28 22:57:45 dist-test-slave-kzfr krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:45.826138 2790 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:57:45.826510 2790 webserver.cc:460] Webserver started at http://127.0.0.1:45671/ using document root <none> and password file <none>
I20260928 22:57:45.826713 2790 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:45.826754 2790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:45.826848 2790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:45.827920 2790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "6155e88b369142b9b51a69d8307883ee"
format_stamp: "Formatted at 2026-09-28 22:57:45 on dist-test-slave-kzfr"
server_key: "2ae483de1a63191fff0969d0054174c8"
server_key_iv: "b9a1fff86bcb28a804a90404e52678f5"
server_key_version: "encryptionkey@0"
I20260928 22:57:45.828220 2790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "6155e88b369142b9b51a69d8307883ee"
format_stamp: "Formatted at 2026-09-28 22:57:45 on dist-test-slave-kzfr"
server_key: "2ae483de1a63191fff0969d0054174c8"
server_key_iv: "b9a1fff86bcb28a804a90404e52678f5"
server_key_version: "encryptionkey@0"
I20260928 22:57:45.829461 2790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:57:45.830255 2805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:45.830430 2790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:45.830521 2790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "6155e88b369142b9b51a69d8307883ee"
format_stamp: "Formatted at 2026-09-28 22:57:45 on dist-test-slave-kzfr"
server_key: "2ae483de1a63191fff0969d0054174c8"
server_key_iv: "b9a1fff86bcb28a804a90404e52678f5"
server_key_version: "encryptionkey@0"
I20260928 22:57:45.830603 2790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:45.852540 2790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:45.854965 2790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:45.855091 2790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:45.858938 2790 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44023
I20260928 22:57:45.858971 2857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44023 every 8 connection(s)
I20260928 22:57:45.859360 2790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:45.860258 2858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:45.862600 2858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee: Bootstrap starting.
I20260928 22:57:45.863200 2858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:45.863492 2858 log.cc:824] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:45.863755 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 2790
I20260928 22:57:45.863893 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:45.864106 14394 external_mini_cluster.cc:1428] Setting key 00cea9f430493335d52343fa2f6b5ee2
I20260928 22:57:45.864311 2858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee: No bootstrap required, opened a new log
I20260928 22:57:45.866608 2858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6155e88b369142b9b51a69d8307883ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44023 } }
I20260928 22:57:45.866822 2858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:45.866894 2858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6155e88b369142b9b51a69d8307883ee, State: Initialized, Role: FOLLOWER
I20260928 22:57:45.867051 2858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6155e88b369142b9b51a69d8307883ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44023 } }
I20260928 22:57:45.867131 2858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:45.867161 2858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:45.867208 2858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:45.867882 2858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6155e88b369142b9b51a69d8307883ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44023 } }
I20260928 22:57:45.867992 2858 leader_election.cc:302] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6155e88b369142b9b51a69d8307883ee; no voters:
I20260928 22:57:45.868131 2858 leader_election.cc:288] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:45.868191 2863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:45.868363 2858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:45.868379 2863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [term 1 LEADER]: Becoming Leader. State: Replica: 6155e88b369142b9b51a69d8307883ee, State: Running, Role: LEADER
I20260928 22:57:45.868579 2863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6155e88b369142b9b51a69d8307883ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44023 } }
I20260928 22:57:45.869668 2864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6155e88b369142b9b51a69d8307883ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6155e88b369142b9b51a69d8307883ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44023 } } }
I20260928 22:57:45.869801 2864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [sys.catalog]: This master's current role is: LEADER
W20260928 22:57:45.870249 2878 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:57:45.870890 2878 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:57:45.869719 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6155e88b369142b9b51a69d8307883ee. Latest consensus state: current_term: 1 leader_uuid: "6155e88b369142b9b51a69d8307883ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6155e88b369142b9b51a69d8307883ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44023 } } }
I20260928 22:57:45.871034 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:45.871174 2879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:45.871363 2879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:57:45 dist-test-slave-kzfr krb5kdc[2777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 22:57:45.874068 2879 catalog_manager.cc:1347] Generated new cluster ID: 62ca95d73b7c4d478e250f6a55b0597f
I20260928 22:57:45.874410 2879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:45.876636 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:45.864731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:44673 (local address 127.0.0.1:44023)
0928 22:57:45.864951 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:45.864962 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:45.865002 (+ 40us) server_negotiation.cc:407] Connection header received
0928 22:57:45.865819 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:45.865829 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:45.865895 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:45.866044 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:45.866624 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:45.868327 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:45.869783 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:45.870379 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:45.870489 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:45.873278 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:45.873281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:45.873281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:45.873291 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:45.875074 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:45.875476 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:45.875480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:45.875480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:45.875571 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:45.875817 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:45.875818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:45.875818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:45.876052 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:45.876251 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:45.876338 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 22:57:45.876435 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":110,"threads_started":1}
I20260928 22:57:45.877565 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:45.891753 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:45.881882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:60685 (local address 127.0.0.1:44023)
0928 22:57:45.881922 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:45.881924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:45.885760 (+ 3836us) server_negotiation.cc:407] Connection header received
0928 22:57:45.885826 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:45.885827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:45.885840 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:45.885909 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:45.886428 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:45.886918 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:45.887660 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:45.887881 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:45.887957 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:45.889124 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:45.889125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:45.889126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:45.889133 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:45.890565 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:45.890946 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:45.890947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:45.890947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:45.891005 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:45.891235 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:45.891236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:45.891237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:45.891347 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:45.891457 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:45.891581 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 22:57:45.891644 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:57:45.893918 2879 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:45.894461 2879 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:45.900651 2879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6155e88b369142b9b51a69d8307883ee: Generated new TSK 0
I20260928 22:57:45.900856 2879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:45.903350 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 2790
W20260928 22:57:45.907821 2775 connection.cc:535] client connection to 127.0.0.1:44023 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44023): Connection reset by peer (error 104)
2026-09-28T22:57:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:45 dist-test-slave-kzfr krb5kdc[2889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:45 dist-test-slave-kzfr krb5kdc[2889](info): set up 2 sockets
Sep 28 22:57:45 dist-test-slave-kzfr krb5kdc[2889](info): commencing operation
krb5kdc: starting...
W20260928 22:57:46.990374 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:47 dist-test-slave-kzfr krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:57:47.117071 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:47.200580 2902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:47.200800 2902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:47.200826 2902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:47.202031 2902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:47.202075 2902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:47.202090 2902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:47.202103 2902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:47.202116 2902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:47.203548 2902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:47.203755 2902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:47.203936 2902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:47.208902 2910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:47.208917 2907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:47.208963 2908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:47.209331 2902 server_base.cc:1034] running on GCE node
I20260928 22:57:47.209656 2902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:47.209983 2902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:47.211164 2902 hybrid_clock.cc:648] HybridClock initialized: now 1790636267211150 us; error 48 us; skew 500 ppm
Sep 28 22:57:47 dist-test-slave-kzfr krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:47.213877 2902 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:57:47.214339 2902 webserver.cc:460] Webserver started at http://127.0.0.1:33965/ using document root <none> and password file <none>
I20260928 22:57:47.214601 2902 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:47.214672 2902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:47.214826 2902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:47.215914 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "792bcdd84811401b8bbe978b042e70ca"
format_stamp: "Formatted at 2026-09-28 22:57:47 on dist-test-slave-kzfr"
server_key: "9b315d1c05ea59945f291a93e88a8491"
server_key_iv: "a25ee3198e0d44105d6284170f622821"
server_key_version: "encryptionkey@0"
I20260928 22:57:47.216284 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "792bcdd84811401b8bbe978b042e70ca"
format_stamp: "Formatted at 2026-09-28 22:57:47 on dist-test-slave-kzfr"
server_key: "9b315d1c05ea59945f291a93e88a8491"
server_key_iv: "a25ee3198e0d44105d6284170f622821"
server_key_version: "encryptionkey@0"
I20260928 22:57:47.217767 2902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:57:47.218590 2917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:47.218784 2902 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:47.218902 2902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "792bcdd84811401b8bbe978b042e70ca"
format_stamp: "Formatted at 2026-09-28 22:57:47 on dist-test-slave-kzfr"
server_key: "9b315d1c05ea59945f291a93e88a8491"
server_key_iv: "a25ee3198e0d44105d6284170f622821"
server_key_version: "encryptionkey@0"
I20260928 22:57:47.218984 2902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:47.226651 2902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:47.228924 2902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:47.229085 2902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:47.233186 2902 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35523
I20260928 22:57:47.233218 2969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35523 every 8 connection(s)
I20260928 22:57:47.233566 2902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:47.234517 2970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:47.236745 2970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca: Bootstrap starting.
I20260928 22:57:47.237275 2970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:47.237509 2970 log.cc:824] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:47.238245 2970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca: No bootstrap required, opened a new log
I20260928 22:57:47.239820 2970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792bcdd84811401b8bbe978b042e70ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35523 } }
I20260928 22:57:47.239964 2970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:47.239986 2970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792bcdd84811401b8bbe978b042e70ca, State: Initialized, Role: FOLLOWER
I20260928 22:57:47.240125 2970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792bcdd84811401b8bbe978b042e70ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35523 } }
I20260928 22:57:47.240209 2970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:47.240232 2970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:47.240262 2970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:47.240777 2970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792bcdd84811401b8bbe978b042e70ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35523 } }
I20260928 22:57:47.240862 2970 leader_election.cc:302] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 792bcdd84811401b8bbe978b042e70ca; no voters:
I20260928 22:57:47.240985 2970 leader_election.cc:288] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:47.241060 2973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:47.241217 2973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [term 1 LEADER]: Becoming Leader. State: Replica: 792bcdd84811401b8bbe978b042e70ca, State: Running, Role: LEADER
I20260928 22:57:47.241257 2970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:47.241307 2973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792bcdd84811401b8bbe978b042e70ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35523 } }
I20260928 22:57:47.241698 2975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 792bcdd84811401b8bbe978b042e70ca. Latest consensus state: current_term: 1 leader_uuid: "792bcdd84811401b8bbe978b042e70ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792bcdd84811401b8bbe978b042e70ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35523 } } }
I20260928 22:57:47.241844 2975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:47.241854 2974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "792bcdd84811401b8bbe978b042e70ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792bcdd84811401b8bbe978b042e70ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35523 } } }
I20260928 22:57:47.241987 2974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:47.242268 2982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:47.242667 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 2902
I20260928 22:57:47.242786 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:47.242870 2982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:47.242980 14394 external_mini_cluster.cc:1428] Setting key b11b77362fc073be750330b9c2a0aebb
I20260928 22:57:47.244556 2982 catalog_manager.cc:1347] Generated new cluster ID: 18ab111cf6ce434b8f1e0a1d6a378846
I20260928 22:57:47.244607 2982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:47.250952 2992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:47.243432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:46323 (local address 127.0.0.1:35523)
0928 22:57:47.243842 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:47.243851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:47.243879 (+ 28us) server_negotiation.cc:407] Connection header received
0928 22:57:47.243992 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:47.243999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:47.244071 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:47.244467 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:47.244560 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:47.245964 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:47.247060 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:47.247709 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:47.247823 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:47.250227 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:47.250232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:47.250232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:47.250435 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:47.250479 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:47.250493 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:47.250577 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:47.250723 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 22:57:47.250752 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":82,"threads_started":1}
I20260928 22:57:47.251901 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:47.257097 2982 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:47.257211 2992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:47.253308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:34903 (local address 127.0.0.1:35523)
0928 22:57:47.253346 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:47.253347 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:47.253615 (+ 268us) server_negotiation.cc:407] Connection header received
0928 22:57:47.253676 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:47.253677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:47.253686 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:47.253743 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:47.253891 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:47.254584 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:47.255595 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:47.255917 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:47.255962 (+ 45us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:47.256756 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:47.256757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:47.256758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:47.256762 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:47.256780 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:47.256784 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:47.256848 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:47.257133 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 22:57:47.257147 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":15}
I20260928 22:57:47.257820 2982 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:47.260128 2982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 792bcdd84811401b8bbe978b042e70ca: Generated new TSK 0
I20260928 22:57:47.260298 2982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:47.268590 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 2902
W20260928 22:57:47.272704 2887 connection.cc:535] client connection to 127.0.0.1:35523 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35523): Connection reset by peer (error 104)
2026-09-28T22:57:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:47 dist-test-slave-kzfr krb5kdc[3001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:47 dist-test-slave-kzfr krb5kdc[3001](info): set up 2 sockets
Sep 28 22:57:47 dist-test-slave-kzfr krb5kdc[3001](info): commencing operation
krb5kdc: starting...
W20260928 22:57:48.305969 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:48 dist-test-slave-kzfr krb5kdc[3001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:57:48.424213 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:48.504815 3014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:48.505020 3014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:48.505044 3014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:48.506168 3014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:48.506206 3014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:48.506219 3014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:48.506232 3014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:48.506244 3014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:48.507651 3014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:48.507841 3014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:48.508009 3014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:48.513207 3019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:48.513232 3022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:48.513265 3014 server_base.cc:1034] running on GCE node
W20260928 22:57:48.513192 3020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:48.513724 3014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:48.514017 3014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:48.515189 3014 hybrid_clock.cc:648] HybridClock initialized: now 1790636268515179 us; error 50 us; skew 500 ppm
Sep 28 22:57:48 dist-test-slave-kzfr krb5kdc[3001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:48.523656 3014 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:57:48.524113 3014 webserver.cc:460] Webserver started at http://127.0.0.1:37197/ using document root <none> and password file <none>
I20260928 22:57:48.524370 3014 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:48.524415 3014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:48.524570 3014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:48.525756 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "fc3ac62f540748e3be8d18fc97e8769c"
format_stamp: "Formatted at 2026-09-28 22:57:48 on dist-test-slave-kzfr"
server_key: "b2c0f8e6c516e7e4f7ec172244b0a08e"
server_key_iv: "6c4432747e4b5f8eeeaf88052378d1f9"
server_key_version: "encryptionkey@0"
I20260928 22:57:48.526105 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "fc3ac62f540748e3be8d18fc97e8769c"
format_stamp: "Formatted at 2026-09-28 22:57:48 on dist-test-slave-kzfr"
server_key: "b2c0f8e6c516e7e4f7ec172244b0a08e"
server_key_iv: "6c4432747e4b5f8eeeaf88052378d1f9"
server_key_version: "encryptionkey@0"
I20260928 22:57:48.527436 3014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:57:48.528190 3029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:48.528380 3014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:57:48.528488 3014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "fc3ac62f540748e3be8d18fc97e8769c"
format_stamp: "Formatted at 2026-09-28 22:57:48 on dist-test-slave-kzfr"
server_key: "b2c0f8e6c516e7e4f7ec172244b0a08e"
server_key_iv: "6c4432747e4b5f8eeeaf88052378d1f9"
server_key_version: "encryptionkey@0"
I20260928 22:57:48.528563 3014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:48.531875 3014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:48.533720 3014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:48.533885 3014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:48.537721 3014 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38763
I20260928 22:57:48.537768 3081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38763 every 8 connection(s)
I20260928 22:57:48.538096 3014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:48.538937 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 3014
I20260928 22:57:48.539018 3082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:48.539062 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:48.539264 14394 external_mini_cluster.cc:1428] Setting key 98ead2ccef3ccdceddc63d086e9a8aa4
I20260928 22:57:48.541751 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:48.539802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:58313 (local address 127.0.0.1:38763)
0928 22:57:48.539960 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:48.539976 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:48.540001 (+ 25us) server_negotiation.cc:407] Connection header received
0928 22:57:48.540482 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:48.540488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:48.540514 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:48.540634 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:48.541262 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:48.541265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:48.541265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:48.541269 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:48.541306 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:48.541315 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:48.541372 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:48.541456 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 22:57:48.541524 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260928 22:57:48.542436 3082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c: Bootstrap starting.
I20260928 22:57:48.542866 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:48.543169 3082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:48.543475 3082 log.cc:824] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:48.544256 3082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c: No bootstrap required, opened a new log
I20260928 22:57:48.545321 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:48.544390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:41039 (local address 127.0.0.1:38763)
0928 22:57:48.544447 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:48.544449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:48.544544 (+ 95us) server_negotiation.cc:407] Connection header received
0928 22:57:48.544600 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:48.544601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:48.544604 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:48.544668 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:48.545108 (+ 440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:48.545109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:48.545109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:48.545112 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:48.545123 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:48.545126 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:48.545162 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:48.545225 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 22:57:48.545235 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:57:48.546624 3082 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3ac62f540748e3be8d18fc97e8769c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38763 } }
I20260928 22:57:48.546754 3082 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:48.546777 3082 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc3ac62f540748e3be8d18fc97e8769c, State: Initialized, Role: FOLLOWER
I20260928 22:57:48.546921 3082 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3ac62f540748e3be8d18fc97e8769c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38763 } }
I20260928 22:57:48.547000 3082 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:48.547046 3082 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:48.547096 3082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:48.547624 3082 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3ac62f540748e3be8d18fc97e8769c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38763 } }
I20260928 22:57:48.547734 3082 leader_election.cc:302] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc3ac62f540748e3be8d18fc97e8769c; no voters:
I20260928 22:57:48.547894 3082 leader_election.cc:288] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:48.547968 3092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:48.548148 3092 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [term 1 LEADER]: Becoming Leader. State: Replica: fc3ac62f540748e3be8d18fc97e8769c, State: Running, Role: LEADER
I20260928 22:57:48.548159 3082 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:48.548305 3092 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3ac62f540748e3be8d18fc97e8769c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38763 } }
I20260928 22:57:48.548698 3094 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc3ac62f540748e3be8d18fc97e8769c. Latest consensus state: current_term: 1 leader_uuid: "fc3ac62f540748e3be8d18fc97e8769c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3ac62f540748e3be8d18fc97e8769c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38763 } } }
I20260928 22:57:48.548725 3093 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc3ac62f540748e3be8d18fc97e8769c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3ac62f540748e3be8d18fc97e8769c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38763 } } }
I20260928 22:57:48.548854 3094 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:48.548874 3093 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:48.549059 3101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:48.549525 3101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:48.551174 3101 catalog_manager.cc:1347] Generated new cluster ID: c84c59b120434f7c9c3aca6ce8e8b04f
I20260928 22:57:48.551225 3101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:48.579914 3101 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:48.580114 3101 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:48.584348 3101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc3ac62f540748e3be8d18fc97e8769c: Generated new TSK 0
I20260928 22:57:48.584556 3101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:48.590025 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 3014
2026-09-28T22:57:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:48 dist-test-slave-kzfr krb5kdc[3113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:48 dist-test-slave-kzfr krb5kdc[3113](info): set up 2 sockets
Sep 28 22:57:48 dist-test-slave-kzfr krb5kdc[3113](info): commencing operation
krb5kdc: starting...
W20260928 22:57:49.653949 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:49 dist-test-slave-kzfr krb5kdc[3113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:57:49.772433 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:49.853670 3126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:49.853955 3126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:49.854010 3126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:49.855611 3126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:49.855655 3126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:49.855677 3126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:49.855726 3126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:49.855757 3126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:49.857259 3126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:49.857512 3126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:49.857776 3126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:49.862563 3131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:49.862577 3132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:49.862707 3126 server_base.cc:1034] running on GCE node
W20260928 22:57:49.862565 3134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:49.862993 3126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:49.863337 3126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:49.864501 3126 hybrid_clock.cc:648] HybridClock initialized: now 1790636269864498 us; error 34 us; skew 500 ppm
Sep 28 22:57:49 dist-test-slave-kzfr krb5kdc[3113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:49.866891 3126 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:57:49.867367 3126 webserver.cc:460] Webserver started at http://127.0.0.1:35961/ using document root <none> and password file <none>
I20260928 22:57:49.867607 3126 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:49.867676 3126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:49.867818 3126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:49.868880 3126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "6601cc4da2fd463d84327192edfd5094"
format_stamp: "Formatted at 2026-09-28 22:57:49 on dist-test-slave-kzfr"
server_key: "d712a0f056ff36fbf6508f3ee4534ab2"
server_key_iv: "3969f5647e31183124168c8657a2354b"
server_key_version: "encryptionkey@0"
I20260928 22:57:49.869246 3126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "6601cc4da2fd463d84327192edfd5094"
format_stamp: "Formatted at 2026-09-28 22:57:49 on dist-test-slave-kzfr"
server_key: "d712a0f056ff36fbf6508f3ee4534ab2"
server_key_iv: "3969f5647e31183124168c8657a2354b"
server_key_version: "encryptionkey@0"
I20260928 22:57:49.870637 3126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:57:49.871387 3141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:49.871690 3126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:57:49.871791 3126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "6601cc4da2fd463d84327192edfd5094"
format_stamp: "Formatted at 2026-09-28 22:57:49 on dist-test-slave-kzfr"
server_key: "d712a0f056ff36fbf6508f3ee4534ab2"
server_key_iv: "3969f5647e31183124168c8657a2354b"
server_key_version: "encryptionkey@0"
I20260928 22:57:49.871866 3126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:49.887171 3126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:49.889127 3126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:49.889304 3126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:49.893380 3126 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45775
I20260928 22:57:49.893417 3193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45775 every 8 connection(s)
I20260928 22:57:49.893798 3126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:49.894714 3194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:49.896759 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 3126
I20260928 22:57:49.896893 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:49.897068 14394 external_mini_cluster.cc:1428] Setting key fd388ada7cd51cd1dc7aa514ce796098
I20260928 22:57:49.897085 3194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094: Bootstrap starting.
I20260928 22:57:49.897800 3194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:49.898104 3194 log.cc:824] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:49.899154 3194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094: No bootstrap required, opened a new log
I20260928 22:57:49.901757 3194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6601cc4da2fd463d84327192edfd5094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45775 } }
I20260928 22:57:49.901948 3194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:49.902015 3194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6601cc4da2fd463d84327192edfd5094, State: Initialized, Role: FOLLOWER
I20260928 22:57:49.902187 3194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6601cc4da2fd463d84327192edfd5094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45775 } }
I20260928 22:57:49.902272 3194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:49.902320 3194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:49.902374 3194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:49.903144 3194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6601cc4da2fd463d84327192edfd5094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45775 } }
I20260928 22:57:49.903282 3194 leader_election.cc:302] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6601cc4da2fd463d84327192edfd5094; no voters:
I20260928 22:57:49.903509 3194 leader_election.cc:288] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:49.903533 3199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:49.903663 3199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [term 1 LEADER]: Becoming Leader. State: Replica: 6601cc4da2fd463d84327192edfd5094, State: Running, Role: LEADER
I20260928 22:57:49.903810 3194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:49.903815 3199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6601cc4da2fd463d84327192edfd5094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45775 } }
I20260928 22:57:49.904311 3200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6601cc4da2fd463d84327192edfd5094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6601cc4da2fd463d84327192edfd5094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45775 } } }
I20260928 22:57:49.904680 3200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [sys.catalog]: This master's current role is: LEADER
Sep 28 22:57:49 dist-test-slave-kzfr krb5kdc[3113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 22:57:49.904582 3201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6601cc4da2fd463d84327192edfd5094. Latest consensus state: current_term: 1 leader_uuid: "6601cc4da2fd463d84327192edfd5094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6601cc4da2fd463d84327192edfd5094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45775 } } }
I20260928 22:57:49.905247 3201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:49.905438 3211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:49.906327 3211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:49.908022 3211 catalog_manager.cc:1347] Generated new cluster ID: 69896909f0864377be42ea520602e136
I20260928 22:57:49.908077 3211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:49.909029 3197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:49.897675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:54405 (local address 127.0.0.1:45775)
0928 22:57:49.897863 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:49.897874 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:49.897907 (+ 33us) server_negotiation.cc:407] Connection header received
0928 22:57:49.898400 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:49.898408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:49.898494 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:49.898656 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:49.899592 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:49.900486 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:49.901825 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:49.902372 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:49.902482 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:49.905457 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:49.905461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:49.905461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:49.905474 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:49.907312 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:49.907764 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:49.907767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:49.907768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:49.907857 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:49.908093 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:49.908094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:49.908095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:49.908413 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:49.908573 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:49.908708 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 22:57:49.908812 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
I20260928 22:57:49.910776 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:49.918515 3211 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:49.919216 3211 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:49.921241 3197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:49.913886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:44153 (local address 127.0.0.1:45775)
0928 22:57:49.913927 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:49.913929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:49.913998 (+ 69us) server_negotiation.cc:407] Connection header received
0928 22:57:49.914055 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:49.914055 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:49.914064 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:49.914116 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:49.914744 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:49.915489 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:49.916518 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:49.916844 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:49.916904 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:49.918454 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:49.918455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:49.918456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:49.918463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:49.920054 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:49.920471 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:49.920472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:49.920472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:49.920524 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:49.920750 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:49.920751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:49.920751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:49.920860 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:49.920957 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:49.921091 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 22:57:49.921147 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:57:49.924084 3211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6601cc4da2fd463d84327192edfd5094: Generated new TSK 0
I20260928 22:57:49.924275 3211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:49.929093 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 3126
W20260928 22:57:49.932910 3111 connection.cc:535] client connection to 127.0.0.1:45775 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45775): Connection reset by peer (error 104)
2026-09-28T22:57:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:49 dist-test-slave-kzfr krb5kdc[3225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:49 dist-test-slave-kzfr krb5kdc[3225](info): set up 2 sockets
Sep 28 22:57:49 dist-test-slave-kzfr krb5kdc[3225](info): commencing operation
krb5kdc: starting...
W20260928 22:57:51.010083 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:51 dist-test-slave-kzfr krb5kdc[3225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:57:51.134820 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:51.218274 3238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:51.218500 3238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:51.218524 3238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:51.219641 3238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:51.219678 3238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:51.219692 3238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:51.219705 3238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:51.219717 3238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:51.221165 3238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:51.221378 3238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:51.221624 3238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:51.226744 3244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:51.226763 3243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:51.226833 3238 server_base.cc:1034] running on GCE node
W20260928 22:57:51.226778 3246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:51.227283 3238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:51.227622 3238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:51.228785 3238 hybrid_clock.cc:648] HybridClock initialized: now 1790636271228780 us; error 39 us; skew 500 ppm
Sep 28 22:57:51 dist-test-slave-kzfr krb5kdc[3225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:51.231355 3238 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:57:51.231840 3238 webserver.cc:460] Webserver started at http://127.0.0.1:36867/ using document root <none> and password file <none>
I20260928 22:57:51.232077 3238 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:51.232155 3238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:51.232309 3238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:51.233359 3238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "339c0754efeb48e9b5e76f02803a4d8c"
format_stamp: "Formatted at 2026-09-28 22:57:51 on dist-test-slave-kzfr"
server_key: "05ba249cf4434d82c62fd885e601ef6e"
server_key_iv: "be4a4c06fd92b0b79ea6cd8165cf723c"
server_key_version: "encryptionkey@0"
I20260928 22:57:51.233767 3238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "339c0754efeb48e9b5e76f02803a4d8c"
format_stamp: "Formatted at 2026-09-28 22:57:51 on dist-test-slave-kzfr"
server_key: "05ba249cf4434d82c62fd885e601ef6e"
server_key_iv: "be4a4c06fd92b0b79ea6cd8165cf723c"
server_key_version: "encryptionkey@0"
I20260928 22:57:51.235075 3238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:57:51.235816 3253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:51.235992 3238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:57:51.236089 3238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "339c0754efeb48e9b5e76f02803a4d8c"
format_stamp: "Formatted at 2026-09-28 22:57:51 on dist-test-slave-kzfr"
server_key: "05ba249cf4434d82c62fd885e601ef6e"
server_key_iv: "be4a4c06fd92b0b79ea6cd8165cf723c"
server_key_version: "encryptionkey@0"
I20260928 22:57:51.236193 3238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:51.248052 3238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:51.250022 3238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:51.250203 3238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:51.254362 3238 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32893
I20260928 22:57:51.254400 3305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32893 every 8 connection(s)
I20260928 22:57:51.254786 3238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:51.255712 3306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:51.258064 3306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c: Bootstrap starting.
I20260928 22:57:51.258677 3306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:51.259016 3306 log.cc:824] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:51.259519 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 3238
I20260928 22:57:51.259641 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:51.259764 3306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c: No bootstrap required, opened a new log
I20260928 22:57:51.259871 14394 external_mini_cluster.cc:1428] Setting key 2f900eb6de6967a8ec05f2afcc2bc544
I20260928 22:57:51.261780 3306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339c0754efeb48e9b5e76f02803a4d8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32893 } }
I20260928 22:57:51.261948 3306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:51.261988 3306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 339c0754efeb48e9b5e76f02803a4d8c, State: Initialized, Role: FOLLOWER
I20260928 22:57:51.262193 3306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339c0754efeb48e9b5e76f02803a4d8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32893 } }
I20260928 22:57:51.262283 3306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:51.262308 3306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:51.262383 3306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:51.262938 3306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339c0754efeb48e9b5e76f02803a4d8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32893 } }
I20260928 22:57:51.263023 3306 leader_election.cc:302] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 339c0754efeb48e9b5e76f02803a4d8c; no voters:
I20260928 22:57:51.263202 3306 leader_election.cc:288] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:51.263250 3311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:51.263412 3311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [term 1 LEADER]: Becoming Leader. State: Replica: 339c0754efeb48e9b5e76f02803a4d8c, State: Running, Role: LEADER
I20260928 22:57:51.263460 3306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:51.263590 3311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339c0754efeb48e9b5e76f02803a4d8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32893 } }
I20260928 22:57:51.265432 3312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "339c0754efeb48e9b5e76f02803a4d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339c0754efeb48e9b5e76f02803a4d8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32893 } } }
W20260928 22:57:51.265440 3325 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 22:57:51.265498 3312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [sys.catalog]: This master's current role is: LEADER
W20260928 22:57:51.265511 3325 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:57:51.265199 3313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 339c0754efeb48e9b5e76f02803a4d8c. Latest consensus state: current_term: 1 leader_uuid: "339c0754efeb48e9b5e76f02803a4d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339c0754efeb48e9b5e76f02803a4d8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32893 } } }
I20260928 22:57:51.265647 3313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:51.265695 3327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:51.265847 3327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:51.267429 3327 catalog_manager.cc:1347] Generated new cluster ID: 048a7a1100474c2c9e6d76f5e542d759
I20260928 22:57:51.267475 3327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:51.267678 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:51.260421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:37541 (local address 127.0.0.1:32893)
0928 22:57:51.260590 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:51.260606 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:51.260637 (+ 31us) server_negotiation.cc:407] Connection header received
0928 22:57:51.261765 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:51.261772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:51.261828 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:51.261967 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:51.262218 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:51.263808 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:51.265048 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:51.265760 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:51.265878 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:51.266973 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:51.266976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:51.266977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:51.267157 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:51.267218 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:51.267230 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:51.267331 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:51.267414 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 22:57:51.267445 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260928 22:57:51.268584 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:51.274367 3327 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:51.275077 3327 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:51.276907 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:51.270006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:37713 (local address 127.0.0.1:32893)
0928 22:57:51.270043 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:51.270045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:51.273862 (+ 3817us) server_negotiation.cc:407] Connection header received
0928 22:57:51.273941 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:51.273942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:51.273953 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:51.274022 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:51.274200 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:51.274781 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:51.275603 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:51.275837 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:51.275915 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:51.276627 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:51.276628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:51.276628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:51.276633 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:51.276647 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:51.276651 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:51.276717 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:51.276800 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 22:57:51.276814 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:57:51.280922 3327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 339c0754efeb48e9b5e76f02803a4d8c: Generated new TSK 0
I20260928 22:57:51.281075 3327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:51.290480 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 3238
W20260928 22:57:51.294770 3223 connection.cc:535] client connection to 127.0.0.1:32893 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32893): Connection reset by peer (error 104)
2026-09-28T22:57:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:51 dist-test-slave-kzfr krb5kdc[3337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:51 dist-test-slave-kzfr krb5kdc[3337](info): set up 2 sockets
Sep 28 22:57:51 dist-test-slave-kzfr krb5kdc[3337](info): commencing operation
krb5kdc: starting...
W20260928 22:57:52.373993 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:52 dist-test-slave-kzfr krb5kdc[3337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:57:52.495870 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:52.576318 3350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:52.576581 3350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:52.576623 3350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:52.577795 3350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:52.577872 3350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:52.577895 3350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:52.577924 3350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:52.577955 3350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:52.579443 3350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42409
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:52.579739 3350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:52.579972 3350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:52.585182 3355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:52.585206 3358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:52.585318 3350 server_base.cc:1034] running on GCE node
W20260928 22:57:52.585201 3356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:52.585788 3350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:52.586092 3350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:52.587258 3350 hybrid_clock.cc:648] HybridClock initialized: now 1790636272587239 us; error 42 us; skew 500 ppm
Sep 28 22:57:52 dist-test-slave-kzfr krb5kdc[3337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:52.589809 3350 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:57:52.590317 3350 webserver.cc:460] Webserver started at http://127.0.0.1:41425/ using document root <none> and password file <none>
I20260928 22:57:52.590564 3350 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:52.590634 3350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:52.590787 3350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:52.591882 3350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "33ef5bbc97d443748803fef74d65265c"
format_stamp: "Formatted at 2026-09-28 22:57:52 on dist-test-slave-kzfr"
server_key: "62431c1097b70a577ffd54dc53bfe2f2"
server_key_iv: "e598696380f92aa1c58fd1fa3002c6ba"
server_key_version: "encryptionkey@0"
I20260928 22:57:52.592262 3350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "33ef5bbc97d443748803fef74d65265c"
format_stamp: "Formatted at 2026-09-28 22:57:52 on dist-test-slave-kzfr"
server_key: "62431c1097b70a577ffd54dc53bfe2f2"
server_key_iv: "e598696380f92aa1c58fd1fa3002c6ba"
server_key_version: "encryptionkey@0"
I20260928 22:57:52.593657 3350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 22:57:52.594375 3365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:52.594580 3350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:52.594688 3350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "33ef5bbc97d443748803fef74d65265c"
format_stamp: "Formatted at 2026-09-28 22:57:52 on dist-test-slave-kzfr"
server_key: "62431c1097b70a577ffd54dc53bfe2f2"
server_key_iv: "e598696380f92aa1c58fd1fa3002c6ba"
server_key_version: "encryptionkey@0"
I20260928 22:57:52.594770 3350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:52.598115 3350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:52.600498 3350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:52.600687 3350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:52.604565 3350 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42409
I20260928 22:57:52.604697 3417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42409 every 8 connection(s)
I20260928 22:57:52.604983 3350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:52.605901 3418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:52.608134 3418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c: Bootstrap starting.
I20260928 22:57:52.608695 3418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:52.608973 3418 log.cc:824] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:52.609709 3418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c: No bootstrap required, opened a new log
I20260928 22:57:52.610742 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 3350
I20260928 22:57:52.610862 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:52.611042 14394 external_mini_cluster.cc:1428] Setting key 4869363abd9d207d55d77ef67995c8d8
I20260928 22:57:52.611858 3418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ef5bbc97d443748803fef74d65265c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42409 } }
I20260928 22:57:52.612025 3418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:52.612064 3418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ef5bbc97d443748803fef74d65265c, State: Initialized, Role: FOLLOWER
I20260928 22:57:52.612279 3418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ef5bbc97d443748803fef74d65265c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42409 } }
I20260928 22:57:52.612360 3418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:52.612406 3418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:52.612488 3418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:52.613219 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:52.611545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:37777 (local address 127.0.0.1:42409)
0928 22:57:52.611735 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:52.611742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:52.611768 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:57:52.611862 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:52.611867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:52.611890 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:52.611996 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:52.612707 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:52.612710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:52.612711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:52.612715 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:52.612762 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:52.612776 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:52.612854 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:52.612914 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 22:57:52.612959 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260928 22:57:52.613755 3418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ef5bbc97d443748803fef74d65265c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42409 } }
I20260928 22:57:52.613921 3418 leader_election.cc:302] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33ef5bbc97d443748803fef74d65265c; no voters:
I20260928 22:57:52.614087 3418 leader_election.cc:288] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:52.614131 3423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:52.614262 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:52.614292 3423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [term 1 LEADER]: Becoming Leader. State: Replica: 33ef5bbc97d443748803fef74d65265c, State: Running, Role: LEADER
I20260928 22:57:52.614333 3418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:52.614372 3423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ef5bbc97d443748803fef74d65265c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42409 } }
I20260928 22:57:52.614663 3424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33ef5bbc97d443748803fef74d65265c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ef5bbc97d443748803fef74d65265c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42409 } } }
I20260928 22:57:52.614758 3424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:52.614919 3425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33ef5bbc97d443748803fef74d65265c. Latest consensus state: current_term: 1 leader_uuid: "33ef5bbc97d443748803fef74d65265c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ef5bbc97d443748803fef74d65265c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42409 } } }
I20260928 22:57:52.614962 3429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:52.614993 3425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:52.615581 3429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:52.617213 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:52.615917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:40619 (local address 127.0.0.1:42409)
0928 22:57:52.615960 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:52.615962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:52.616212 (+ 250us) server_negotiation.cc:407] Connection header received
0928 22:57:52.616287 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:52.616287 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:52.616291 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:52.616374 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:52.616998 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:52.616999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:52.616999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:52.617004 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:52.617016 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:52.617020 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:52.617066 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:52.617135 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 22:57:52.617148 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:57:52.617369 3429 catalog_manager.cc:1347] Generated new cluster ID: f55e581c15c64c59a0d0738598c564fb
I20260928 22:57:52.617417 3429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:52.642167 3429 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:52.642401 3429 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:52.649972 3429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33ef5bbc97d443748803fef74d65265c: Generated new TSK 0
I20260928 22:57:52.650189 3429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:52.662120 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 3350
2026-09-28T22:57:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:52 dist-test-slave-kzfr krb5kdc[3449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:52 dist-test-slave-kzfr krb5kdc[3449](info): set up 2 sockets
Sep 28 22:57:52 dist-test-slave-kzfr krb5kdc[3449](info): commencing operation
krb5kdc: starting...
W20260928 22:57:53.689980 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:53 dist-test-slave-kzfr krb5kdc[3449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:57:53.813398 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:40003
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:41969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:40003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:53.896735 3462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:53.896957 3462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:53.896981 3462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:53.898204 3462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:53.898247 3462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:53.898262 3462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:53.898274 3462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:53.898288 3462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:53.899777 3462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:41969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:40003
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:40003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:53.900032 3462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:53.900281 3462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:57:53.905282 3462 server_base.cc:1034] running on GCE node
W20260928 22:57:53.905200 3467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:53.905200 3470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:53.905233 3468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:53.905735 3462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:53.906025 3462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:53.907195 3462 hybrid_clock.cc:648] HybridClock initialized: now 1790636273907183 us; error 43 us; skew 500 ppm
Sep 28 22:57:53 dist-test-slave-kzfr krb5kdc[3449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636273, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:53.913681 3462 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:57:53.914150 3462 webserver.cc:460] Webserver started at http://127.14.14.190:38015/ using document root <none> and password file <none>
I20260928 22:57:53.914376 3462 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:53.914443 3462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:53.914599 3462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:53.915678 3462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "8d639f5807fa4eb6b721c2836bccd423"
format_stamp: "Formatted at 2026-09-28 22:57:53 on dist-test-slave-kzfr"
server_key: "ff96066c6a120798fe50ac8cd8d65f4f"
server_key_iv: "a13bba7ab01c0631f6b0cae09bbfb234"
server_key_version: "encryptionkey@0"
I20260928 22:57:53.916044 3462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "8d639f5807fa4eb6b721c2836bccd423"
format_stamp: "Formatted at 2026-09-28 22:57:53 on dist-test-slave-kzfr"
server_key: "ff96066c6a120798fe50ac8cd8d65f4f"
server_key_iv: "a13bba7ab01c0631f6b0cae09bbfb234"
server_key_version: "encryptionkey@0"
I20260928 22:57:53.917362 3462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:57:53.918183 3477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:53.918368 3462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:53.918464 3462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "8d639f5807fa4eb6b721c2836bccd423"
format_stamp: "Formatted at 2026-09-28 22:57:53 on dist-test-slave-kzfr"
server_key: "ff96066c6a120798fe50ac8cd8d65f4f"
server_key_iv: "a13bba7ab01c0631f6b0cae09bbfb234"
server_key_version: "encryptionkey@0"
I20260928 22:57:53.918551 3462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:53.939107 3462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:53.941157 3462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:53.941323 3462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:53.945461 3462 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:40003
I20260928 22:57:53.945516 3529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:40003 every 8 connection(s)
I20260928 22:57:53.945917 3462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:53.946862 3530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:53.948220 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 3462
I20260928 22:57:53.948342 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:53.948532 14394 external_mini_cluster.cc:1428] Setting key d5bc2c4640382db2d47a86a6f2fc7565
I20260928 22:57:53.949731 3530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423: Bootstrap starting.
I20260928 22:57:53.950467 3530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:53.950834 3530 log.cc:824] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:53.951586 3530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423: No bootstrap required, opened a new log
I20260928 22:57:53.954267 3530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d639f5807fa4eb6b721c2836bccd423" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 40003 } }
I20260928 22:57:53.954448 3530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:53.954489 3530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d639f5807fa4eb6b721c2836bccd423, State: Initialized, Role: FOLLOWER
I20260928 22:57:53.954651 3530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d639f5807fa4eb6b721c2836bccd423" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 40003 } }
I20260928 22:57:53.954747 3530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:53.954787 3530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:53.954842 3530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:53.955536 3530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d639f5807fa4eb6b721c2836bccd423" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 40003 } }
I20260928 22:57:53.955639 3530 leader_election.cc:302] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d639f5807fa4eb6b721c2836bccd423; no voters:
Sep 28 22:57:53 dist-test-slave-kzfr krb5kdc[3449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:57:53.955852 3530 leader_election.cc:288] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:53.955883 3535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:53.956075 3535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [term 1 LEADER]: Becoming Leader. State: Replica: 8d639f5807fa4eb6b721c2836bccd423, State: Running, Role: LEADER
I20260928 22:57:53.956144 3530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:53.956214 3535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d639f5807fa4eb6b721c2836bccd423" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 40003 } }
I20260928 22:57:53.956652 3536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d639f5807fa4eb6b721c2836bccd423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d639f5807fa4eb6b721c2836bccd423" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 40003 } } }
I20260928 22:57:53.956734 3536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:53.956955 3537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d639f5807fa4eb6b721c2836bccd423. Latest consensus state: current_term: 1 leader_uuid: "8d639f5807fa4eb6b721c2836bccd423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d639f5807fa4eb6b721c2836bccd423" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 40003 } } }
I20260928 22:57:53.957083 3537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:53.957257 3542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:53.958016 3542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:53.959578 3542 catalog_manager.cc:1347] Generated new cluster ID: 4eeb47bb505b4dbabb3f30ad22118b46
I20260928 22:57:53.959631 3542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:53.960281 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:53.949053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35845 (local address 127.14.14.190:40003)
0928 22:57:53.949243 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:53.949251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:53.949277 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:57:53.949732 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:53.949738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:53.949806 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:53.949942 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:53.950772 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:53.951769 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:53.952994 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:53.953383 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:53.953460 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:53.956388 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:53.956393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:53.956393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:53.956410 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:53.958518 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:53.958973 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:53.958976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:53.958977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:53.959090 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:53.959317 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:53.959319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:53.959319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:53.959669 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:53.959874 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:53.960009 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 22:57:53.960109 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260928 22:57:53.962591 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:53.972803 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:53.965937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43429 (local address 127.14.14.190:40003)
0928 22:57:53.965978 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:53.965980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:53.966212 (+ 232us) server_negotiation.cc:407] Connection header received
0928 22:57:53.966288 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:53.966289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:53.966300 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:53.966365 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:53.967087 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:53.967582 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:53.968324 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:53.968561 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:53.968636 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:53.969959 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:53.969961 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:53.969961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:53.969968 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:53.971517 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:53.971891 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:53.971893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:53.971893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:53.971958 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:53.972190 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:53.972190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:53.972191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:53.972330 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:53.972491 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:53.972618 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 22:57:53.972683 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":20}
I20260928 22:57:53.994488 3542 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:53.995050 3542 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:54.001557 3542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d639f5807fa4eb6b721c2836bccd423: Generated new TSK 0
I20260928 22:57:54.001794 3542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:54.005087 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 3462
W20260928 22:57:54.009511 3447 connection.cc:535] client connection to 127.14.14.190:40003 recv error: Network error: failed to read from TLS socket (remote: 127.14.14.190:40003): Connection reset by peer (error 104)
2026-09-28T22:57:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:54 dist-test-slave-kzfr krb5kdc[3561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:54 dist-test-slave-kzfr krb5kdc[3561](info): set up 2 sockets
Sep 28 22:57:54 dist-test-slave-kzfr krb5kdc[3561](info): commencing operation
krb5kdc: starting...
W20260928 22:57:55.130000 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:55 dist-test-slave-kzfr krb5kdc[3561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:57:55.249331 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:37739
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:37739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:55.329205 3574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:55.329412 3574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:55.329434 3574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:55.330646 3574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:55.330694 3574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:55.330709 3574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:55.330722 3574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:55.330734 3574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:55.332232 3574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:37739
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:37739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:55.332499 3574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:55.332736 3574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:55.337644 3582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:55.337709 3580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:55.337700 3579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:55.337880 3574 server_base.cc:1034] running on GCE node
I20260928 22:57:55.338110 3574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:55.338456 3574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:55.339613 3574 hybrid_clock.cc:648] HybridClock initialized: now 1790636275339588 us; error 46 us; skew 500 ppm
Sep 28 22:57:55 dist-test-slave-kzfr krb5kdc[3561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636275, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:55.342201 3574 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:57:55.342684 3574 webserver.cc:460] Webserver started at http://127.14.14.190:34225/ using document root <none> and password file <none>
I20260928 22:57:55.342916 3574 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:55.342985 3574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:55.343153 3574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:55.344607 3574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "b5828dceab3c4dc5a6f51ed0c893a525"
format_stamp: "Formatted at 2026-09-28 22:57:55 on dist-test-slave-kzfr"
server_key: "208151ec0338d906bad493b2dfe7ec54"
server_key_iv: "5e2ff3fa43510a6053d4f3595e1bfc5e"
server_key_version: "encryptionkey@0"
I20260928 22:57:55.344986 3574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "b5828dceab3c4dc5a6f51ed0c893a525"
format_stamp: "Formatted at 2026-09-28 22:57:55 on dist-test-slave-kzfr"
server_key: "208151ec0338d906bad493b2dfe7ec54"
server_key_iv: "5e2ff3fa43510a6053d4f3595e1bfc5e"
server_key_version: "encryptionkey@0"
I20260928 22:57:55.346405 3574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:57:55.347124 3589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:55.347302 3574 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:55.347407 3574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "b5828dceab3c4dc5a6f51ed0c893a525"
format_stamp: "Formatted at 2026-09-28 22:57:55 on dist-test-slave-kzfr"
server_key: "208151ec0338d906bad493b2dfe7ec54"
server_key_iv: "5e2ff3fa43510a6053d4f3595e1bfc5e"
server_key_version: "encryptionkey@0"
I20260928 22:57:55.347481 3574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:55.362556 3574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:55.364444 3574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:55.364606 3574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:55.368791 3574 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:37739
I20260928 22:57:55.368821 3641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:37739 every 8 connection(s)
I20260928 22:57:55.369191 3574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:55.370123 3642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:55.372500 3642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525: Bootstrap starting.
I20260928 22:57:55.373101 3642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:55.373401 3642 log.cc:824] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:55.374181 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 3574
I20260928 22:57:55.374267 3642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525: No bootstrap required, opened a new log
I20260928 22:57:55.374305 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:55.374486 14394 external_mini_cluster.cc:1428] Setting key 0aab7bc62912f32c90feb998f5cdc67e
I20260928 22:57:55.376948 3642 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5828dceab3c4dc5a6f51ed0c893a525" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37739 } }
I20260928 22:57:55.377162 3642 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:55.377202 3642 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5828dceab3c4dc5a6f51ed0c893a525, State: Initialized, Role: FOLLOWER
I20260928 22:57:55.377355 3642 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5828dceab3c4dc5a6f51ed0c893a525" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37739 } }
I20260928 22:57:55.377439 3642 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:55.377468 3642 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:55.377509 3642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:55.378248 3642 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5828dceab3c4dc5a6f51ed0c893a525" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37739 } }
I20260928 22:57:55.378337 3642 leader_election.cc:302] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5828dceab3c4dc5a6f51ed0c893a525; no voters:
I20260928 22:57:55.378537 3642 leader_election.cc:288] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:55.378590 3647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:55.378738 3647 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [term 1 LEADER]: Becoming Leader. State: Replica: b5828dceab3c4dc5a6f51ed0c893a525, State: Running, Role: LEADER
I20260928 22:57:55.378803 3642 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:55.378978 3647 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5828dceab3c4dc5a6f51ed0c893a525" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37739 } }
I20260928 22:57:55.379307 3647 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5828dceab3c4dc5a6f51ed0c893a525. Latest consensus state: current_term: 1 leader_uuid: "b5828dceab3c4dc5a6f51ed0c893a525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5828dceab3c4dc5a6f51ed0c893a525" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37739 } } }
I20260928 22:57:55.379400 3647 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:55.379315 3648 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5828dceab3c4dc5a6f51ed0c893a525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5828dceab3c4dc5a6f51ed0c893a525" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37739 } } }
I20260928 22:57:55.379522 3648 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:55.380282 3654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:55.380888 3646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:55.375016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34193 (local address 127.14.14.190:37739)
0928 22:57:55.375214 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:55.375222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:55.375250 (+ 28us) server_negotiation.cc:407] Connection header received
0928 22:57:55.375632 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:55.375639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:55.375697 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:55.375808 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:55.376041 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:55.376933 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:55.378082 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:55.378679 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:55.379121 (+ 442us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:55.380109 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:55.380113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:55.380114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:55.380346 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:55.380382 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:55.380396 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:55.380481 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:55.380598 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 22:57:55.380629 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260928 22:57:55.380899 3654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:55.381942 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:55.382970 3654 catalog_manager.cc:1347] Generated new cluster ID: dd1265a83f6d4cbd9748fa2d02d4823a
I20260928 22:57:55.383015 3654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:55.386417 3646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:55.383431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37375 (local address 127.14.14.190:37739)
0928 22:57:55.383466 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:55.383468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:55.383591 (+ 123us) server_negotiation.cc:407] Connection header received
0928 22:57:55.383653 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:55.383654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:55.383664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:55.383720 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:55.383853 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:55.384321 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:55.385064 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:55.385301 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:55.385390 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:55.386126 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:55.386128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:55.386128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:55.386132 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:55.386146 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:55.386150 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:55.386190 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:55.386308 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 22:57:55.386317 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:57:55.390375 3654 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:55.391076 3654 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:55.401924 3654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5828dceab3c4dc5a6f51ed0c893a525: Generated new TSK 0
I20260928 22:57:55.402120 3654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:55.405390 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 3574
W20260928 22:57:55.410169 3559 connection.cc:535] client connection to 127.14.14.190:37739 recv error: Network error: failed to read from TLS socket (remote: 127.14.14.190:37739): Connection reset by peer (error 104)
2026-09-28T22:57:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:55 dist-test-slave-kzfr krb5kdc[3673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:55 dist-test-slave-kzfr krb5kdc[3673](info): set up 2 sockets
Sep 28 22:57:55 dist-test-slave-kzfr krb5kdc[3673](info): commencing operation
krb5kdc: starting...
W20260928 22:57:56.446259 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:56 dist-test-slave-kzfr krb5kdc[3673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:57:56.564291 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:33991
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:33991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:56.645678 3686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:56.645948 3686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:56.645992 3686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:56.647132 3686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:56.647217 3686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:56.647243 3686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:56.647264 3686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:56.647291 3686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:56.648741 3686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:33991
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:33991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:56.649017 3686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:56.649252 3686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:56.654155 3694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:56.654279 3686 server_base.cc:1034] running on GCE node
W20260928 22:57:56.654145 3691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:56.654148 3692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:56.654663 3686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:56.654903 3686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:56.656044 3686 hybrid_clock.cc:648] HybridClock initialized: now 1790636276656025 us; error 39 us; skew 500 ppm
Sep 28 22:57:56 dist-test-slave-kzfr krb5kdc[3673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636276, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:56.658591 3686 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:57:56.659047 3686 webserver.cc:460] Webserver started at http://127.14.14.190:42781/ using document root <none> and password file <none>
I20260928 22:57:56.659283 3686 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:56.659328 3686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:56.659457 3686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:56.660521 3686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "57a52304d75c4c9c8bca4ac87ec0295a"
format_stamp: "Formatted at 2026-09-28 22:57:56 on dist-test-slave-kzfr"
server_key: "bd5fa0b608952bc15fcaeb628e3dce47"
server_key_iv: "6cb321def0d27d357abf68f235ec8eb1"
server_key_version: "encryptionkey@0"
I20260928 22:57:56.660892 3686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "57a52304d75c4c9c8bca4ac87ec0295a"
format_stamp: "Formatted at 2026-09-28 22:57:56 on dist-test-slave-kzfr"
server_key: "bd5fa0b608952bc15fcaeb628e3dce47"
server_key_iv: "6cb321def0d27d357abf68f235ec8eb1"
server_key_version: "encryptionkey@0"
I20260928 22:57:56.662321 3686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:57:56.663091 3701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:56.663286 3686 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:56.663357 3686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "57a52304d75c4c9c8bca4ac87ec0295a"
format_stamp: "Formatted at 2026-09-28 22:57:56 on dist-test-slave-kzfr"
server_key: "bd5fa0b608952bc15fcaeb628e3dce47"
server_key_iv: "6cb321def0d27d357abf68f235ec8eb1"
server_key_version: "encryptionkey@0"
I20260928 22:57:56.663448 3686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:56.666805 3686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:56.668718 3686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:56.668851 3686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:56.672917 3686 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:33991
I20260928 22:57:56.672947 3753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:33991 every 8 connection(s)
I20260928 22:57:56.673296 3686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:56.674503 3754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:56.676806 3754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a: Bootstrap starting.
I20260928 22:57:56.677417 3754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:56.677758 3754 log.cc:824] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:56.678431 3754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a: No bootstrap required, opened a new log
I20260928 22:57:56.679132 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 3686
I20260928 22:57:56.679265 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:56.679458 14394 external_mini_cluster.cc:1428] Setting key 97758a9c22bf01eb75e0c148a417e46d
I20260928 22:57:56.680759 3754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a52304d75c4c9c8bca4ac87ec0295a" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33991 } }
I20260928 22:57:56.680930 3754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:56.680967 3754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a52304d75c4c9c8bca4ac87ec0295a, State: Initialized, Role: FOLLOWER
I20260928 22:57:56.681419 3754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a52304d75c4c9c8bca4ac87ec0295a" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33991 } }
I20260928 22:57:56.681608 3754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:56.681663 3754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:56.681725 3754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:56.682085 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:56.680001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53869 (local address 127.14.14.190:33991)
0928 22:57:56.680191 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:56.680201 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:56.680240 (+ 39us) server_negotiation.cc:407] Connection header received
0928 22:57:56.680717 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:56.680726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:56.680762 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:56.680917 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:56.681563 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:56.681567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:56.681567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:56.681572 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:56.681643 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:56.681659 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:56.681726 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:56.681813 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 22:57:56.681854 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260928 22:57:56.682910 3754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a52304d75c4c9c8bca4ac87ec0295a" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33991 } }
I20260928 22:57:56.683018 3754 leader_election.cc:302] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57a52304d75c4c9c8bca4ac87ec0295a; no voters:
I20260928 22:57:56.683081 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:56.683184 3754 leader_election.cc:288] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:56.683216 3759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:56.683326 3759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [term 1 LEADER]: Becoming Leader. State: Replica: 57a52304d75c4c9c8bca4ac87ec0295a, State: Running, Role: LEADER
I20260928 22:57:56.683422 3754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:56.683454 3759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a52304d75c4c9c8bca4ac87ec0295a" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33991 } }
I20260928 22:57:56.683794 3760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57a52304d75c4c9c8bca4ac87ec0295a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a52304d75c4c9c8bca4ac87ec0295a" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33991 } } }
I20260928 22:57:56.683923 3760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:56.683779 3759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57a52304d75c4c9c8bca4ac87ec0295a. Latest consensus state: current_term: 1 leader_uuid: "57a52304d75c4c9c8bca4ac87ec0295a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a52304d75c4c9c8bca4ac87ec0295a" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33991 } } }
I20260928 22:57:56.684106 3759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:56.684262 3767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:56.684778 3767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:56.687011 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:56.685841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33573 (local address 127.14.14.190:33991)
0928 22:57:56.685878 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:56.685880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:56.685973 (+ 93us) server_negotiation.cc:407] Connection header received
0928 22:57:56.686017 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:56.686018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:56.686024 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:56.686090 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:56.686768 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:56.686769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:56.686770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:56.686774 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:56.686788 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:56.686792 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:56.686849 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:56.686928 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 22:57:56.686943 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:57:56.687479 3767 catalog_manager.cc:1347] Generated new cluster ID: 1d1d4f90b4e54472bae39b7b427b921e
I20260928 22:57:56.687527 3767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:56.703274 3767 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:56.703465 3767 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:56.710947 3767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57a52304d75c4c9c8bca4ac87ec0295a: Generated new TSK 0
I20260928 22:57:56.711161 3767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:56.721984 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 3686
2026-09-28T22:57:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:56 dist-test-slave-kzfr krb5kdc[3785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:56 dist-test-slave-kzfr krb5kdc[3785](info): set up 2 sockets
Sep 28 22:57:56 dist-test-slave-kzfr krb5kdc[3785](info): commencing operation
krb5kdc: starting...
W20260928 22:57:57.757923 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:57 dist-test-slave-kzfr krb5kdc[3785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:57:57.880501 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:34687
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:36669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:34687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:57.961851 3798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:57.962075 3798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:57.962100 3798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:57.963254 3798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:57.963299 3798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:57.963313 3798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:57.963326 3798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:57.963387 3798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:57.964849 3798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:36669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:34687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:34687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:57.965117 3798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:57.965363 3798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:57.970448 3804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:57.970528 3803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:57.970448 3806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:57.970702 3798 server_base.cc:1034] running on GCE node
I20260928 22:57:57.970947 3798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:57.971287 3798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:57.972474 3798 hybrid_clock.cc:648] HybridClock initialized: now 1790636277972452 us; error 51 us; skew 500 ppm
Sep 28 22:57:57 dist-test-slave-kzfr krb5kdc[3785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636277, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:57.975160 3798 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:57:57.975562 3798 webserver.cc:460] Webserver started at http://127.14.14.190:36507/ using document root <none> and password file <none>
I20260928 22:57:57.975768 3798 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:57.975811 3798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:57.975899 3798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:57.976922 3798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "f6113734747f4e25ba5e13ee9afa4cd8"
format_stamp: "Formatted at 2026-09-28 22:57:57 on dist-test-slave-kzfr"
server_key: "2d718a732a0ed49e47da881f40863429"
server_key_iv: "b03338fc3ad7b461e31fe3c75bcc7289"
server_key_version: "encryptionkey@0"
I20260928 22:57:57.977221 3798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "f6113734747f4e25ba5e13ee9afa4cd8"
format_stamp: "Formatted at 2026-09-28 22:57:57 on dist-test-slave-kzfr"
server_key: "2d718a732a0ed49e47da881f40863429"
server_key_iv: "b03338fc3ad7b461e31fe3c75bcc7289"
server_key_version: "encryptionkey@0"
I20260928 22:57:57.978706 3798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:57:57.979384 3813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:57.979576 3798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:57.979672 3798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "f6113734747f4e25ba5e13ee9afa4cd8"
format_stamp: "Formatted at 2026-09-28 22:57:57 on dist-test-slave-kzfr"
server_key: "2d718a732a0ed49e47da881f40863429"
server_key_iv: "b03338fc3ad7b461e31fe3c75bcc7289"
server_key_version: "encryptionkey@0"
I20260928 22:57:57.979722 3798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:57.987236 3798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:57.988926 3798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:57.989061 3798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:57.993251 3798 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:34687
I20260928 22:57:57.993280 3865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:34687 every 8 connection(s)
I20260928 22:57:57.993659 3798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:57.994546 3866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:57.995237 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 3798
I20260928 22:57:57.995355 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:57.995553 14394 external_mini_cluster.cc:1428] Setting key 075ba0590024feb46df0a2356aac1e03
I20260928 22:57:57.997749 3866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8: Bootstrap starting.
I20260928 22:57:57.998472 3866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:57.998780 3866 log.cc:824] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:57.999508 3866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8: No bootstrap required, opened a new log
I20260928 22:57:58.001758 3866 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6113734747f4e25ba5e13ee9afa4cd8" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 34687 } }
I20260928 22:57:58.001892 3866 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:58.001914 3866 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6113734747f4e25ba5e13ee9afa4cd8, State: Initialized, Role: FOLLOWER
I20260928 22:57:58.002070 3866 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6113734747f4e25ba5e13ee9afa4cd8" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 34687 } }
I20260928 22:57:58.002132 3866 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:58.002154 3866 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:58.002221 3866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 22:57:57 dist-test-slave-kzfr krb5kdc[3785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:57:58.002948 3866 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6113734747f4e25ba5e13ee9afa4cd8" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 34687 } }
I20260928 22:57:58.003069 3866 leader_election.cc:302] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6113734747f4e25ba5e13ee9afa4cd8; no voters:
I20260928 22:57:58.003262 3866 leader_election.cc:288] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:58.003290 3871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:58.003399 3871 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [term 1 LEADER]: Becoming Leader. State: Replica: f6113734747f4e25ba5e13ee9afa4cd8, State: Running, Role: LEADER
I20260928 22:57:58.003547 3871 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6113734747f4e25ba5e13ee9afa4cd8" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 34687 } }
I20260928 22:57:58.003531 3866 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:58.004148 3872 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6113734747f4e25ba5e13ee9afa4cd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6113734747f4e25ba5e13ee9afa4cd8" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 34687 } } }
I20260928 22:57:58.004383 3872 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:58.004317 3873 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6113734747f4e25ba5e13ee9afa4cd8. Latest consensus state: current_term: 1 leader_uuid: "f6113734747f4e25ba5e13ee9afa4cd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6113734747f4e25ba5e13ee9afa4cd8" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 34687 } } }
I20260928 22:57:58.004774 3873 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:58.005343 3880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:58.005959 3880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:58.007717 3880 catalog_manager.cc:1347] Generated new cluster ID: e55748dd5a254ad6b6225cb13b704898
I20260928 22:57:58.007778 3880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:58.007942 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:57.996099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50649 (local address 127.14.14.190:34687)
0928 22:57:57.996286 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:57.996296 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:57.996330 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:57:57.996870 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:57.996878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:57.996941 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:57.997099 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:57.997780 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:57.999188 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:57.999986 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:58.000625 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:58.000752 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:58.003498 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:58.003502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:58.003502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:58.003514 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:58.005786 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:58.006187 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:58.006189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:58.006190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:58.006300 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:58.006522 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:58.006523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:58.006523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:58.006831 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:58.006972 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:58.007502 (+ 530us) server_negotiation.cc:299] Negotiation successful
0928 22:57:58.007656 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260928 22:57:58.008828 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:58.017884 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:58.010364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46005 (local address 127.14.14.190:34687)
0928 22:57:58.010398 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:58.010400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:58.010444 (+ 44us) server_negotiation.cc:407] Connection header received
0928 22:57:58.010498 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:58.010498 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:58.010508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:58.010582 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:58.011161 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:58.012310 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:58.013271 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:58.013643 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:58.013720 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:58.015026 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:58.015027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:58.015028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:57:58.015035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:58.016667 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:58.017040 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:58.017041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:58.017042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:58.017104 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:57:58.017309 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:57:58.017310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:57:58.017310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:57:58.017443 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:58.017551 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:58.017667 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 22:57:58.017747 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:57:58.020015 3880 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:58.020553 3880 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:58.027350 3880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6113734747f4e25ba5e13ee9afa4cd8: Generated new TSK 0
I20260928 22:57:58.027554 3880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:58.030617 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 3798
2026-09-28T22:57:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:58 dist-test-slave-kzfr krb5kdc[3897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:58 dist-test-slave-kzfr krb5kdc[3897](info): set up 2 sockets
Sep 28 22:57:58 dist-test-slave-kzfr krb5kdc[3897](info): commencing operation
krb5kdc: starting...
W20260928 22:57:59.061986 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:57:59 dist-test-slave-kzfr krb5kdc[3897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:57:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:57:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:57:59.184495 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:37917
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:46037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:37917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:57:59.267180 3910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:57:59.267393 3910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:57:59.267416 3910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:57:59.268806 3910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:57:59.268868 3910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:57:59.268884 3910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:57:59.268898 3910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:57:59.268951 3910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:57:59.270565 3910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:46037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:37917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:37917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:57:59.270817 3910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:57:59.271060 3910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:57:59.276204 3915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:59.276355 3910 server_base.cc:1034] running on GCE node
W20260928 22:57:59.276207 3916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:57:59.276204 3918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:57:59.276731 3910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:57:59.277004 3910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:57:59.278198 3910 hybrid_clock.cc:648] HybridClock initialized: now 1790636279278176 us; error 34 us; skew 500 ppm
Sep 28 22:57:59 dist-test-slave-kzfr krb5kdc[3897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636279, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:57:59.287828 3910 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:57:59.288332 3910 webserver.cc:460] Webserver started at http://127.14.14.190:40603/ using document root <none> and password file <none>
I20260928 22:57:59.288590 3910 fs_manager.cc:359] Metadata directory not provided
I20260928 22:57:59.288661 3910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:57:59.288818 3910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:57:59.290026 3910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "0f36698552494b3a8e3ba68d60d64573"
format_stamp: "Formatted at 2026-09-28 22:57:59 on dist-test-slave-kzfr"
server_key: "55571087f4e8764109be891d06a9fde6"
server_key_iv: "dce320f6bb7db6e4610aa5b2a398306f"
server_key_version: "encryptionkey@0"
I20260928 22:57:59.290403 3910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "0f36698552494b3a8e3ba68d60d64573"
format_stamp: "Formatted at 2026-09-28 22:57:59 on dist-test-slave-kzfr"
server_key: "55571087f4e8764109be891d06a9fde6"
server_key_iv: "dce320f6bb7db6e4610aa5b2a398306f"
server_key_version: "encryptionkey@0"
I20260928 22:57:59.291769 3910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:57:59.292513 3925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:57:59.292690 3910 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:57:59.292800 3910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "0f36698552494b3a8e3ba68d60d64573"
format_stamp: "Formatted at 2026-09-28 22:57:59 on dist-test-slave-kzfr"
server_key: "55571087f4e8764109be891d06a9fde6"
server_key_iv: "dce320f6bb7db6e4610aa5b2a398306f"
server_key_version: "encryptionkey@0"
I20260928 22:57:59.292876 3910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:57:59.300752 3910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:57:59.302604 3910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:57:59.302769 3910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:57:59.306831 3910 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:37917
I20260928 22:57:59.306867 3977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:37917 every 8 connection(s)
I20260928 22:57:59.307253 3910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:57:59.308179 3978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:57:59.309091 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 3910
I20260928 22:57:59.309237 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:57:59.309422 14394 external_mini_cluster.cc:1428] Setting key 7f7d3aaddec25c6b2394a3372c83d7cc
I20260928 22:57:59.311532 3978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573: Bootstrap starting.
I20260928 22:57:59.312345 3978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573: Neither blocks nor log segments found. Creating new log.
I20260928 22:57:59.312695 3978 log.cc:824] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573: Log is configured to *not* fsync() on all Append() calls
I20260928 22:57:59.313457 3978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573: No bootstrap required, opened a new log
I20260928 22:57:59.315357 3981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:59.310000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51697 (local address 127.14.14.190:37917)
0928 22:57:59.310172 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:59.310183 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:59.310213 (+ 30us) server_negotiation.cc:407] Connection header received
0928 22:57:59.310625 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:59.310632 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:59.310680 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:59.310790 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:59.311073 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:59.312035 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:59.313250 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:59.313743 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:59.313823 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:59.314536 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:59.314540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:59.314540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:59.314720 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:59.314751 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:59.314762 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:59.314812 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:59.314927 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 22:57:59.314980 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260928 22:57:59.315483 3978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f36698552494b3a8e3ba68d60d64573" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37917 } }
I20260928 22:57:59.315706 3978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:57:59.315748 3978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f36698552494b3a8e3ba68d60d64573, State: Initialized, Role: FOLLOWER
I20260928 22:57:59.315878 3978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f36698552494b3a8e3ba68d60d64573" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37917 } }
I20260928 22:57:59.315986 3978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:57:59.316031 3978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:57:59.316092 3978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:57:59.316345 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:57:59.316797 3978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f36698552494b3a8e3ba68d60d64573" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37917 } }
I20260928 22:57:59.316917 3978 leader_election.cc:302] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f36698552494b3a8e3ba68d60d64573; no voters:
I20260928 22:57:59.317092 3978 leader_election.cc:288] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:57:59.317126 3983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:57:59.317242 3983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [term 1 LEADER]: Becoming Leader. State: Replica: 0f36698552494b3a8e3ba68d60d64573, State: Running, Role: LEADER
I20260928 22:57:59.317363 3978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:57:59.317396 3983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f36698552494b3a8e3ba68d60d64573" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37917 } }
I20260928 22:57:59.317790 3984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f36698552494b3a8e3ba68d60d64573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f36698552494b3a8e3ba68d60d64573" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37917 } } }
I20260928 22:57:59.317881 3984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:59.318034 3984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f36698552494b3a8e3ba68d60d64573. Latest consensus state: current_term: 1 leader_uuid: "0f36698552494b3a8e3ba68d60d64573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f36698552494b3a8e3ba68d60d64573" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 37917 } } }
I20260928 22:57:59.318094 3994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:57:59.318115 3984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573 [sys.catalog]: This master's current role is: LEADER
I20260928 22:57:59.318619 3994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:57:59.320331 3994 catalog_manager.cc:1347] Generated new cluster ID: cf8a0869f4334b32bb2a133f37a0b7e3
I20260928 22:57:59.320381 3994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:57:59.322484 3981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:57:59.317786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38923 (local address 127.14.14.190:37917)
0928 22:57:59.317818 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 22:57:59.317820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:57:59.319146 (+ 1326us) server_negotiation.cc:407] Connection header received
0928 22:57:59.319231 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:57:59.319232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:57:59.319242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:57:59.319291 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:57:59.319470 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:59.320112 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:59.321172 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:57:59.321466 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:57:59.321540 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:57:59.322257 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:57:59.322258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:57:59.322259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:57:59.322263 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:57:59.322277 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:57:59.322281 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:57:59.322331 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:57:59.322395 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 22:57:59.322415 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 22:57:59.332729 3994 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:57:59.333545 3994 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:57:59.343072 3994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f36698552494b3a8e3ba68d60d64573: Generated new TSK 0
I20260928 22:57:59.343294 3994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:57:59.350082 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 3910
2026-09-28T22:57:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:57:59 dist-test-slave-kzfr krb5kdc[4009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:57:59 dist-test-slave-kzfr krb5kdc[4009](info): set up 2 sockets
Sep 28 22:57:59 dist-test-slave-kzfr krb5kdc[4009](info): commencing operation
krb5kdc: starting...
W20260928 22:58:00.466038 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:58:00 dist-test-slave-kzfr krb5kdc[4009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:58:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:58:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:58:00.586125 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:41517
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:41517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:58:00.667531 4022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:58:00.667753 4022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:58:00.667778 4022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:58:00.668972 4022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:58:00.669018 4022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:58:00.669031 4022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:58:00.669044 4022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:58:00.669057 4022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:58:00.670473 4022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:41517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:41517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:58:00.670667 4022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:58:00.670845 4022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:58:00.675752 4027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:58:00.675799 4030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:58:00.675791 4028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:58:00.675822 4022 server_base.cc:1034] running on GCE node
I20260928 22:58:00.676178 4022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:58:00.676466 4022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:58:00.677652 4022 hybrid_clock.cc:648] HybridClock initialized: now 1790636280677673 us; error 78 us; skew 500 ppm
Sep 28 22:58:00 dist-test-slave-kzfr krb5kdc[4009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636280, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:58:00.680246 4022 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:58:00.680708 4022 webserver.cc:460] Webserver started at http://127.14.14.190:42631/ using document root <none> and password file <none>
I20260928 22:58:00.680943 4022 fs_manager.cc:359] Metadata directory not provided
I20260928 22:58:00.681012 4022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:58:00.681164 4022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:58:00.682317 4022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "7895d59759b9427fa4aa3b073291c15c"
format_stamp: "Formatted at 2026-09-28 22:58:00 on dist-test-slave-kzfr"
server_key: "0931c6ee0e80332a8404a1dd05a6a541"
server_key_iv: "bb1a4b6cd53ecfc75dbcfa1f7b4f51f5"
server_key_version: "encryptionkey@0"
I20260928 22:58:00.682683 4022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "7895d59759b9427fa4aa3b073291c15c"
format_stamp: "Formatted at 2026-09-28 22:58:00 on dist-test-slave-kzfr"
server_key: "0931c6ee0e80332a8404a1dd05a6a541"
server_key_iv: "bb1a4b6cd53ecfc75dbcfa1f7b4f51f5"
server_key_version: "encryptionkey@0"
I20260928 22:58:00.684065 4022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:58:00.684893 4037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:58:00.685098 4022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:58:00.685206 4022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "7895d59759b9427fa4aa3b073291c15c"
format_stamp: "Formatted at 2026-09-28 22:58:00 on dist-test-slave-kzfr"
server_key: "0931c6ee0e80332a8404a1dd05a6a541"
server_key_iv: "bb1a4b6cd53ecfc75dbcfa1f7b4f51f5"
server_key_version: "encryptionkey@0"
I20260928 22:58:00.685286 4022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:58:00.688746 4022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:58:00.690765 4022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:58:00.690925 4022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:58:00.694955 4022 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:41517
I20260928 22:58:00.695000 4089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:41517 every 8 connection(s)
I20260928 22:58:00.695361 4022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:58:00.696346 4090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:58:00.698726 4090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c: Bootstrap starting.
I20260928 22:58:00.699256 4090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c: Neither blocks nor log segments found. Creating new log.
I20260928 22:58:00.699533 4090 log.cc:824] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:58:00.700173 4090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c: No bootstrap required, opened a new log
I20260928 22:58:00.701049 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 4022
I20260928 22:58:00.701164 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:58:00.701350 14394 external_mini_cluster.cc:1428] Setting key 231becc424aa1900ae2e8bf72f8c8f6b
I20260928 22:58:00.702505 4090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7895d59759b9427fa4aa3b073291c15c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 41517 } }
I20260928 22:58:00.702690 4090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:58:00.702733 4090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7895d59759b9427fa4aa3b073291c15c, State: Initialized, Role: FOLLOWER
I20260928 22:58:00.702906 4090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7895d59759b9427fa4aa3b073291c15c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 41517 } }
I20260928 22:58:00.703028 4090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:58:00.703053 4090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:58:00.703181 4090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:58:00.703948 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:00.701945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54425 (local address 127.14.14.190:41517)
0928 22:58:00.702146 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:00.702154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:00.702178 (+ 24us) server_negotiation.cc:407] Connection header received
0928 22:58:00.702486 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:00.702492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:00.702516 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:00.702630 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:00.703447 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:00.703451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:00.703451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:58:00.703456 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:00.703506 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:58:00.703523 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:00.703611 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:00.703647 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 22:58:00.703696 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
I20260928 22:58:00.704319 4090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7895d59759b9427fa4aa3b073291c15c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 41517 } }
I20260928 22:58:00.704423 4090 leader_election.cc:302] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7895d59759b9427fa4aa3b073291c15c; no voters:
I20260928 22:58:00.704581 4090 leader_election.cc:288] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:58:00.704620 4095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:58:00.704775 4095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [term 1 LEADER]: Becoming Leader. State: Replica: 7895d59759b9427fa4aa3b073291c15c, State: Running, Role: LEADER
I20260928 22:58:00.704888 4090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:58:00.704910 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:58:00.704898 4095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7895d59759b9427fa4aa3b073291c15c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 41517 } }
I20260928 22:58:00.705279 4095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7895d59759b9427fa4aa3b073291c15c. Latest consensus state: current_term: 1 leader_uuid: "7895d59759b9427fa4aa3b073291c15c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7895d59759b9427fa4aa3b073291c15c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 41517 } } }
I20260928 22:58:00.705410 4095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [sys.catalog]: This master's current role is: LEADER
I20260928 22:58:00.705897 4102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:58:00.706418 4102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:58:00.706521 4096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7895d59759b9427fa4aa3b073291c15c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7895d59759b9427fa4aa3b073291c15c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 41517 } } }
I20260928 22:58:00.707093 4096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c [sys.catalog]: This master's current role is: LEADER
I20260928 22:58:00.708117 4102 catalog_manager.cc:1347] Generated new cluster ID: 009b30c968a840af8be2d21e447c5ea6
I20260928 22:58:00.708238 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:00.706816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40551 (local address 127.14.14.190:41517)
0928 22:58:00.706857 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:00.706860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:00.707142 (+ 282us) server_negotiation.cc:407] Connection header received
0928 22:58:00.707247 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:00.707248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:00.707253 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:00.707305 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:00.707952 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:00.707953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:00.707953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:58:00.707958 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:00.707973 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:58:00.707977 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:00.708035 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:00.708137 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 22:58:00.708153 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:58:00.708173 4102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:58:00.725221 4102 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:58:00.725427 4102 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:58:00.732016 4102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7895d59759b9427fa4aa3b073291c15c: Generated new TSK 0
I20260928 22:58:00.732218 4102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:58:00.734902 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 4022
2026-09-28T22:58:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1384 ms)
[----------] 18 tests from AuthTokenIssuingTest (24206 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:58:00 dist-test-slave-kzfr krb5kdc[4121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:58:00 dist-test-slave-kzfr krb5kdc[4121](info): set up 2 sockets
Sep 28 22:58:00 dist-test-slave-kzfr krb5kdc[4121](info): commencing operation
krb5kdc: starting...
W20260928 22:58:01.762111 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:58:01 dist-test-slave-kzfr krb5kdc[4121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:58:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:58:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:58:01.884501 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:33313
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:33313,127.14.14.189:42499,127.14.14.188:39617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:58:01.964972 4134 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:58:01.965251 4134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:58:01.965283 4134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:58:01.965325 4134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:58:01.966480 4134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:58:01.966553 4134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:58:01.966578 4134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:58:01.966598 4134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:58:01.966627 4134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:58:01.968104 4134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:33313,127.14.14.189:42499,127.14.14.188:39617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:33313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:58:01.968392 4134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:58:01.968626 4134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:58:01.973903 4139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:58:01.973910 4142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:58:01.973927 4134 server_base.cc:1034] running on GCE node
W20260928 22:58:01.973910 4140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:58:01.974331 4134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:58:01.974623 4134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:58:01.975805 4134 hybrid_clock.cc:648] HybridClock initialized: now 1790636281975785 us; error 38 us; skew 500 ppm
Sep 28 22:58:01 dist-test-slave-kzfr krb5kdc[4121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:58:01.978246 4134 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:58:01.978747 4134 webserver.cc:460] Webserver started at http://127.14.14.190:35201/ using document root <none> and password file <none>
I20260928 22:58:01.978979 4134 fs_manager.cc:359] Metadata directory not provided
I20260928 22:58:01.979050 4134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:58:01.979205 4134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:58:01.980276 4134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "51241efd82074caf8bd53b752838868c"
format_stamp: "Formatted at 2026-09-28 22:58:01 on dist-test-slave-kzfr"
server_key: "3888c4f87208e54cadb37c34021db740"
server_key_iv: "35ffcab25171f5217019a0de0ef25e01"
server_key_version: "encryptionkey@0"
I20260928 22:58:01.980664 4134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "51241efd82074caf8bd53b752838868c"
format_stamp: "Formatted at 2026-09-28 22:58:01 on dist-test-slave-kzfr"
server_key: "3888c4f87208e54cadb37c34021db740"
server_key_iv: "35ffcab25171f5217019a0de0ef25e01"
server_key_version: "encryptionkey@0"
I20260928 22:58:01.982086 4134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:58:01.982846 4149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:58:01.983017 4134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:58:01.983121 4134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "51241efd82074caf8bd53b752838868c"
format_stamp: "Formatted at 2026-09-28 22:58:01 on dist-test-slave-kzfr"
server_key: "3888c4f87208e54cadb37c34021db740"
server_key_iv: "35ffcab25171f5217019a0de0ef25e01"
server_key_version: "encryptionkey@0"
I20260928 22:58:01.983201 4134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:58:01.991473 4134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:58:01.993270 4134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:58:01.993429 4134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:58:01.997635 4201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:33313 every 8 connection(s)
I20260928 22:58:01.997654 4134 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:33313
I20260928 22:58:01.998041 4134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:58:01.998989 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 4134
I20260928 22:58:01.999074 4202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:58:01.999105 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:58:01.999302 14394 external_mini_cluster.cc:1428] Setting key 12a2eed25822cf668799561e28379d6a
I20260928 22:58:02.001317 4202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.14.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.189@KRBTEST.COM" created.
Sep 28 22:58:02 dist-test-slave-kzfr krb5kdc[4121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:58:02.013850 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.002102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41706 (local address 127.14.14.190:33313)
0928 22:58:02.002306 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.002310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.002414 (+ 104us) server_negotiation.cc:407] Connection header received
0928 22:58:02.002946 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.002948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.002967 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.003081 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.003613 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.004425 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.005303 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.005687 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.005811 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.010520 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:02.010528 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:02.010528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:02.010535 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:02.012175 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.012607 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.012609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.012610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.012693 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.012917 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.012918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.012919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.013136 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:02.013241 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:02.013434 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 22:58:02.013570 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"spinlock_wait_cycles":9088,"thread_start_us":119,"threads_started":1}
I20260928 22:58:02.013849 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.002076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33313 (local address 127.0.0.1:41706)
0928 22:58:02.002254 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.002301 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:02.002792 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:02.003088 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:02.003090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:02.003366 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:02.003514 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.003518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.004536 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.004538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.005202 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.005203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.005796 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.005797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.005864 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.006402 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:02.006410 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:02.010413 (+ 4003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:02.012302 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.012309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.012311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.012508 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.012760 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.012761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.012762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.012851 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.013225 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:02.013230 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:02.013292 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 22:58:02.013402 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 22:58:02.013652 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":88,"thread_start_us":87,"threads_started":1}
I20260928 22:58:02.015137 4202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:58:02.015503 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.015393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:42499 (local address 127.0.0.1:51530)
0928 22:58:02.015443 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.015484 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 22:58:02.015642 4153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.14.189:42499: connect: Connection refused (error 111)
I20260928 22:58:02.016041 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.015944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:42499 (local address 127.0.0.1:51540)
0928 22:58:02.015987 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.016011 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 22:58:02.016136 4202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.189:42499: Network error: Client connection negotiation failed: client connection to 127.14.14.189:42499: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.189.keytab.
Entry for principal kudu/127.14.14.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.189.keytab.
I20260928 22:58:02.025477 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.189:42499
--webserver_interface=127.14.14.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.189.keytab
--principal=kudu/127.14.14.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:33313,127.14.14.189:42499,127.14.14.188:39617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:58:02.065371 4202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } attempt: 1
I20260928 22:58:02.065954 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.065811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:42499 (local address 127.0.0.1:51554)
0928 22:58:02.065882 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.065925 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 22:58:02.066349 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.066266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:42499 (local address 127.0.0.1:51564)
0928 22:58:02.066305 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.066331 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 22:58:02.066457 4202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.189:42499: Network error: Client connection negotiation failed: client connection to 127.14.14.189:42499: connect: Connection refused (error 111)
W20260928 22:58:02.106544 4209 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:58:02.106763 4209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:58:02.106782 4209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:58:02.106801 4209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:58:02.107939 4209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:58:02.107981 4209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:58:02.107996 4209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:58:02.108009 4209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:58:02.108021 4209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:58:02.109578 4209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:33313,127.14.14.189:42499,127.14.14.188:39617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.189.keytab
--principal=kudu/127.14.14.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.189:42499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.14.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:58:02.109876 4209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:58:02.110113 4209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:58:02.115062 4214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:58:02.115092 4215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:58:02.115113 4209 server_base.cc:1034] running on GCE node
W20260928 22:58:02.115077 4217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:58:02.115561 4209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:58:02.115846 4209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:58:02.117023 4209 hybrid_clock.cc:648] HybridClock initialized: now 1790636282117009 us; error 43 us; skew 500 ppm
Sep 28 22:58:02 dist-test-slave-kzfr krb5kdc[4121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636282, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:58:02.119468 4209 init.cc:377] Logged in from keytab as kudu/127.14.14.189@KRBTEST.COM (short username kudu)
I20260928 22:58:02.119850 4209 webserver.cc:460] Webserver started at http://127.14.14.189:40835/ using document root <none> and password file <none>
I20260928 22:58:02.120040 4209 fs_manager.cc:359] Metadata directory not provided
I20260928 22:58:02.120079 4209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:58:02.120165 4209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:58:02.121145 4209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/data/instance:
uuid: "8f4c4f6bbe274890b015bf87c2836051"
format_stamp: "Formatted at 2026-09-28 22:58:02 on dist-test-slave-kzfr"
server_key: "f867abc2477b7b5ea66e8f148c45185a"
server_key_iv: "ad753f558deca0684e02685a19b9748d"
server_key_version: "encryptionkey@0"
I20260928 22:58:02.121434 4209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/wal/instance:
uuid: "8f4c4f6bbe274890b015bf87c2836051"
format_stamp: "Formatted at 2026-09-28 22:58:02 on dist-test-slave-kzfr"
server_key: "f867abc2477b7b5ea66e8f148c45185a"
server_key_iv: "ad753f558deca0684e02685a19b9748d"
server_key_version: "encryptionkey@0"
I20260928 22:58:02.122746 4209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:58:02.123410 4224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:58:02.123564 4209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:58:02.123651 4209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/wal
uuid: "8f4c4f6bbe274890b015bf87c2836051"
format_stamp: "Formatted at 2026-09-28 22:58:02 on dist-test-slave-kzfr"
server_key: "f867abc2477b7b5ea66e8f148c45185a"
server_key_iv: "ad753f558deca0684e02685a19b9748d"
server_key_version: "encryptionkey@0"
I20260928 22:58:02.123703 4209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:58:02.134603 4202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } attempt: 2
I20260928 22:58:02.135061 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.134935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:42499 (local address 127.0.0.1:51572)
0928 22:58:02.134996 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.135033 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 22:58:02.135406 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.135319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:42499 (local address 127.0.0.1:51582)
0928 22:58:02.135364 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.135389 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 22:58:02.135504 4202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.189:42499: Network error: Client connection negotiation failed: client connection to 127.14.14.189:42499: connect: Connection refused (error 111)
I20260928 22:58:02.146461 4209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:58:02.148821 4209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:58:02.148969 4209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:58:02.152927 4209 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.189:42499
I20260928 22:58:02.152988 4276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.189:42499 every 8 connection(s)
I20260928 22:58:02.153353 4209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/data/info.pb
I20260928 22:58:02.154387 4277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:58:02.155937 4277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:58:02.160385 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 4209
I20260928 22:58:02.160533 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-1/wal/instance
I20260928 22:58:02.160715 14394 external_mini_cluster.cc:1428] Setting key d24d81e86d5151748c44a53ea66f3270
Sep 28 22:58:02 dist-test-slave-kzfr krb5kdc[4121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636282, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.189@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:58:02.165395 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.156719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.14.14.190:33313)
0928 22:58:02.156813 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.156815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.156908 (+ 93us) server_negotiation.cc:407] Connection header received
0928 22:58:02.157370 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.157371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.157380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.157442 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.158195 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.158710 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.159670 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.159860 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.159915 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.162512 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:02.162513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:02.162513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:02.162520 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:02.163986 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.164407 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.164408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.164408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.164486 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.164733 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.164735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.164735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.164851 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:02.164927 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:02.165222 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 22:58:02.165303 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 22:58:02.165601 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.156601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33313 (local address 127.0.0.1:41714)
0928 22:58:02.156772 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.156815 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:02.157273 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:02.157520 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:02.157524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:02.157835 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:02.158025 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.158029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.158793 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.158794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.159562 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.159564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.159943 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.159944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.160044 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.160577 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:02.160585 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:02.162409 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:02.164102 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.164109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.164111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.164326 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.164556 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.164557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.164558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.164663 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.164932 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:02.164938 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:02.165061 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 22:58:02.165197 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 22:58:02.165348 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260928 22:58:02.166213 4277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.14.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.188@KRBTEST.COM" created.
Sep 28 22:58:02 dist-test-slave-kzfr krb5kdc[4121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636282, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.189@KRBTEST.COM for kudu/127.14.14.189@KRBTEST.COM
I20260928 22:58:02.174813 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.166484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:42499 (local address 127.0.0.1:51596)
0928 22:58:02.166571 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.166581 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:02.166619 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:02.166871 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:02.166872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:02.167011 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:02.167050 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.167051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.167902 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.167903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.168472 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.168474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.168950 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.168951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.169013 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.169425 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:02.169430 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:02.171651 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:02.173725 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.173726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.173727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.173889 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.174144 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.174145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.174145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.174201 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.174485 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:02.174486 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:02.174521 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 22:58:02.174599 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:58:02.174701 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 22:58:02.174898 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.166563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.14.14.189:42499)
0928 22:58:02.166733 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.166737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.166765 (+ 28us) server_negotiation.cc:407] Connection header received
0928 22:58:02.166781 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.166783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.166793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.166871 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.167138 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.167804 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.168558 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.168875 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.168955 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.171749 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:02.171757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:02.171757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:02.171764 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:02.173632 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.173978 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.173981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.173982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.174042 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.174271 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.174272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.174273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.174417 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:02.174505 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:02.174629 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 22:58:02.174770 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260928 22:58:02.175439 4277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:58:02.175818 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.175667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39617 (local address 127.0.0.1:38064)
0928 22:58:02.175720 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.175783 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 22:58:02.175967 4228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
I20260928 22:58:02.176437 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.176317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39617 (local address 127.0.0.1:38072)
0928 22:58:02.176374 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.176405 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 22:58:02.176548 4277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.188:39617: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.188.keytab.
Entry for principal kudu/127.14.14.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.188.keytab.
I20260928 22:58:02.188230 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.188:39617
--webserver_interface=127.14.14.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.188.keytab
--principal=kudu/127.14.14.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:33313,127.14.14.189:42499,127.14.14.188:39617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:58:02.225785 4277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } attempt: 1
I20260928 22:58:02.226553 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.226232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39617 (local address 127.0.0.1:38078)
0928 22:58:02.226372 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.226419 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 22:58:02.226732 4202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } attempt: 3
I20260928 22:58:02.226969 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.226877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39617 (local address 127.0.0.1:38082)
0928 22:58:02.226916 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.226942 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 22:58:02.227065 4277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.188:39617: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
Sep 28 22:58:02 dist-test-slave-kzfr krb5kdc[4121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for kudu/127.14.14.189@KRBTEST.COM
I20260928 22:58:02.234548 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.227032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:42499 (local address 127.0.0.1:51606)
0928 22:58:02.227098 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.227118 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:02.227178 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:02.227332 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:02.227333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:02.227483 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:02.227550 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.227552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.228492 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.228493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.229166 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.229168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.229712 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.229714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.229774 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.230316 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:02.230321 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:02.231822 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:02.233562 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.233563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.233564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.233808 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.234015 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.234015 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.234015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.234067 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.234327 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:02.234328 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:02.234365 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 22:58:02.234413 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:58:02.234483 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:58:02.234594 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.227064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51606 (local address 127.14.14.189:42499)
0928 22:58:02.227110 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.227113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.227182 (+ 69us) server_negotiation.cc:407] Connection header received
0928 22:58:02.227240 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.227241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.227253 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.227306 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.227622 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.228399 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.229254 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.229619 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.229690 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.231913 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:02.231914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:02.231915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:02.231922 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:02.233480 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.233883 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.233884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.233885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.233946 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.234126 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.234127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.234127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.234264 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:02.234332 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:02.234436 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 22:58:02.234511 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:58:02.235117 4202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:58:02.235456 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.235355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39617 (local address 127.0.0.1:38086)
0928 22:58:02.235404 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.235439 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 22:58:02.235745 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.235690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39617 (local address 127.0.0.1:38090)
0928 22:58:02.235713 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.235732 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 22:58:02.235845 4202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.188:39617: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
I20260928 22:58:02.266983 4202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } attempt: 1
I20260928 22:58:02.267474 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.267317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39617 (local address 127.0.0.1:38102)
0928 22:58:02.267386 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.267435 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 22:58:02.267802 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.267718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39617 (local address 127.0.0.1:38116)
0928 22:58:02.267748 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.267777 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 22:58:02.267907 4202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.188:39617: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
W20260928 22:58:02.275290 4284 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:58:02.275488 4284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:58:02.275506 4284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:58:02.275524 4284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:58:02.276589 4284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:58:02.276621 4284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:58:02.276635 4284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:58:02.276648 4284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:58:02.276660 4284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:58:02.278164 4284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:33313,127.14.14.189:42499,127.14.14.188:39617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.188.keytab
--principal=kudu/127.14.14.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.188:39617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.14.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:58:02.278365 4284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:58:02.278585 4284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:58:02.283830 4284 server_base.cc:1034] running on GCE node
W20260928 22:58:02.283859 4289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:58:02.283859 4292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:58:02.283861 4290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:58:02.284353 4284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:58:02.284617 4284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:58:02.285784 4284 hybrid_clock.cc:648] HybridClock initialized: now 1790636282285771 us; error 31 us; skew 500 ppm
Sep 28 22:58:02 dist-test-slave-kzfr krb5kdc[4121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636282, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:58:02.288291 4284 init.cc:377] Logged in from keytab as kudu/127.14.14.188@KRBTEST.COM (short username kudu)
I20260928 22:58:02.288748 4284 webserver.cc:460] Webserver started at http://127.14.14.188:32967/ using document root <none> and password file <none>
I20260928 22:58:02.288945 4284 fs_manager.cc:359] Metadata directory not provided
I20260928 22:58:02.288985 4284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:58:02.289074 4284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:58:02.290201 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/data/instance:
uuid: "b78be04285844e4d82090f93444bbfd7"
format_stamp: "Formatted at 2026-09-28 22:58:02 on dist-test-slave-kzfr"
server_key: "fdb74a9d37d22d6df390d60663d3bb00"
server_key_iv: "a66cf59cee4bc30e27327566bc9c149b"
server_key_version: "encryptionkey@0"
I20260928 22:58:02.290511 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/wal/instance:
uuid: "b78be04285844e4d82090f93444bbfd7"
format_stamp: "Formatted at 2026-09-28 22:58:02 on dist-test-slave-kzfr"
server_key: "fdb74a9d37d22d6df390d60663d3bb00"
server_key_iv: "a66cf59cee4bc30e27327566bc9c149b"
server_key_version: "encryptionkey@0"
I20260928 22:58:02.291746 4284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:58:02.292424 4299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:58:02.292647 4284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:58:02.292743 4284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/wal
uuid: "b78be04285844e4d82090f93444bbfd7"
format_stamp: "Formatted at 2026-09-28 22:58:02 on dist-test-slave-kzfr"
server_key: "fdb74a9d37d22d6df390d60663d3bb00"
server_key_iv: "a66cf59cee4bc30e27327566bc9c149b"
server_key_version: "encryptionkey@0"
I20260928 22:58:02.292800 4284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:58:02.295187 4277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } attempt: 2
I20260928 22:58:02.295571 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.295443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39617 (local address 127.0.0.1:38130)
0928 22:58:02.295503 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.295534 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 22:58:02.295969 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.295883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39617 (local address 127.0.0.1:38142)
0928 22:58:02.295928 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.295952 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 22:58:02.296096 4277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.188:39617: Network error: Client connection negotiation failed: client connection to 127.14.14.188:39617: connect: Connection refused (error 111)
I20260928 22:58:02.306502 4284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:58:02.308743 4284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:58:02.308892 4284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:58:02.313786 4284 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.188:39617
I20260928 22:58:02.313813 4351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.188:39617 every 8 connection(s)
I20260928 22:58:02.314191 4284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/data/info.pb
I20260928 22:58:02.315202 4352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:58:02.316740 4352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:58:02 dist-test-slave-kzfr krb5kdc[4121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636282, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.188@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:58:02.323076 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 4284
I20260928 22:58:02.323210 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790636016256058-14394-0/minicluster-data/master-2/wal/instance
I20260928 22:58:02.323388 14394 external_mini_cluster.cc:1428] Setting key d79d60b71df80747d9bafc2c49f9912a
I20260928 22:58:02.325640 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.317495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41718 (local address 127.14.14.190:33313)
0928 22:58:02.317541 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.317543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.317777 (+ 234us) server_negotiation.cc:407] Connection header received
0928 22:58:02.318215 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.318216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.318225 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.318284 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.318866 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.319353 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.320359 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.320538 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.320588 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.322661 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:02.322662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:02.322662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:02.322668 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:02.324233 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.324636 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.324637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.324638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.324700 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.324954 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.324955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.324955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.325072 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:02.325143 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:02.325414 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 22:58:02.325492 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:58:02.325816 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.317448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:33313 (local address 127.0.0.1:41718)
0928 22:58:02.317640 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.317680 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:02.318128 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:02.318348 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:02.318351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:02.318592 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:02.318795 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.318800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.319427 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.319428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.320288 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.320290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.320614 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.320615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.320732 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.321182 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:02.321190 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:02.322578 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:02.324331 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.324338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.324340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.324559 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.324767 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.324768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.324769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.324879 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.325148 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:02.325154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:02.325285 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 22:58:02.325392 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 22:58:02.325543 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260928 22:58:02.326393 4352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:58:02 dist-test-slave-kzfr krb5kdc[4121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.188@KRBTEST.COM
Sep 28 22:58:02 dist-test-slave-kzfr krb5kdc[4121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636282, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.188@KRBTEST.COM for kudu/127.14.14.189@KRBTEST.COM
I20260928 22:58:02.332544 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.323786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.14.14.188:39617)
0928 22:58:02.324101 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.324105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.324120 (+ 15us) server_negotiation.cc:407] Connection header received
0928 22:58:02.324137 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.324138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.324149 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.324240 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.324847 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.325513 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.326734 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.327170 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.327239 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.329441 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:02.329449 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:02.329449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:02.329456 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:02.331360 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.331764 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.331766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.331766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.331819 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.331988 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.331988 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.331988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.332107 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:02.332276 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:02.332350 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 22:58:02.332428 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":92,"threads_started":1}
I20260928 22:58:02.332908 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.326723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51622 (local address 127.14.14.189:42499)
0928 22:58:02.326771 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.326773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.326781 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:58:02.326805 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.326806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.326816 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.326873 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.327093 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.327660 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.328319 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.328515 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.328589 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.330376 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:02.330378 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:02.330378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:02.330385 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:02.331763 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.332079 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.332080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.332081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.332130 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.332361 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.332362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.332362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.332474 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:02.332551 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:02.332747 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 22:58:02.332825 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:58:02.333189 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.326638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:42499 (local address 127.0.0.1:51622)
0928 22:58:02.326707 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.326719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:02.326760 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:02.326875 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:02.326876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:02.326979 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:02.327023 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.327024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.327747 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.327748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.328234 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.328236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.328586 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.328588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.328655 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.329092 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:02.329099 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:02.330288 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:02.331820 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.331821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.331821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.332027 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.332227 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.332228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.332228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.332283 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.332582 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:02.332583 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:02.332625 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 22:58:02.332993 (+ 368us) client_negotiation.cc:240] Negotiation successful
0928 22:58:02.333087 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:58:02.333639 4352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:58:02 dist-test-slave-kzfr krb5kdc[4121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636282, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.188@KRBTEST.COM for kudu/127.14.14.188@KRBTEST.COM
Sep 28 22:58:02 dist-test-slave-kzfr krb5kdc[4121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:58:02.340123 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.333888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39617 (local address 127.0.0.1:38150)
0928 22:58:02.333937 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.333947 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:02.333984 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:02.334090 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:02.334091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:02.334250 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:02.334277 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.334278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.334898 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.334899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.335336 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.335337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.335728 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.335729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.335779 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.336287 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:02.336291 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:02.337382 (+ 1091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:02.339153 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.339154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.339154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.339320 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.339525 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.339525 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.339526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.339579 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.339822 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:02.339823 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:02.339859 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 22:58:02.339926 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 22:58:02.340017 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 22:58:02.340173 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.333957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.14.14.188:39617)
0928 22:58:02.333993 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.333995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.334005 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:58:02.334032 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.334033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.334041 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.334079 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.334359 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.334819 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.335422 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.335631 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.335711 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.337475 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:02.337476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:02.337476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:02.337482 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:02.339056 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.339405 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.339406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.339406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.339460 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.339649 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.339650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.339651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.339757 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:02.339829 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:02.339953 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 22:58:02.340061 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":21}
I20260928 22:58:02.340703 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.333337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.14.14.190:33313)
0928 22:58:02.333384 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.333386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.333488 (+ 102us) server_negotiation.cc:407] Connection header received
0928 22:58:02.333532 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.333533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.333545 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.333662 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.334225 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.334733 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.335418 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.335621 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.335669 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.337956 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:02.337958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:02.337958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:02.337965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:02.339264 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.339604 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.339605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.339606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.339659 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.339902 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.339903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.339903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.340013 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:02.340538 (+ 525us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:02.340581 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 22:58:02.340637 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:58:02.341828 4352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7: Bootstrap starting.
I20260928 22:58:02.342782 4352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7: Neither blocks nor log segments found. Creating new log.
I20260928 22:58:02.343041 4202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } attempt: 2
I20260928 22:58:02.343091 4352 log.cc:824] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7: Log is configured to *not* fsync() on all Append() calls
I20260928 22:58:02.343858 4352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7: No bootstrap required, opened a new log
Sep 28 22:58:02 dist-test-slave-kzfr krb5kdc[4121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.189@KRBTEST.COM
I20260928 22:58:02.346132 4352 raft_consensus.cc:348] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } }
I20260928 22:58:02.346344 4352 raft_consensus.cc:729] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b78be04285844e4d82090f93444bbfd7, State: Initialized, Role: FOLLOWER
I20260928 22:58:02.346441 4352 consensus_queue.cc:260] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } }
I20260928 22:58:02.346793 4352 sys_catalog.cc:564] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 22:58:02 dist-test-slave-kzfr krb5kdc[4121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for kudu/127.14.14.188@KRBTEST.COM
I20260928 22:58:02.347185 4358 sys_catalog.cc:455] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } } }
I20260928 22:58:02.347294 4358 sys_catalog.cc:458] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:58:02.350246 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.343288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39617 (local address 127.0.0.1:38162)
0928 22:58:02.343333 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.343342 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:02.343382 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:02.343488 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:02.343488 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:02.343625 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:02.343655 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.343656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.344416 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.344417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.344934 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.344936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.345319 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.345320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.345373 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.345848 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:02.345854 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:02.347155 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:02.349128 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.349129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.349130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.349353 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.349560 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.349561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.349561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.349648 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.349932 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:02.349933 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:02.349979 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 22:58:02.350047 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 22:58:02.350142 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 22:58:02.350244 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.341350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.14.14.189:42499)
0928 22:58:02.341384 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.341386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.341394 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:58:02.341444 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.341445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.341453 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.341495 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.342071 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.342503 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.343481 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.343788 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.343835 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.346687 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:02.346688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:02.346689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:02.346696 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:02.348636 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.349015 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.349016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.349017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.349071 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.349293 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.349293 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.349294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.349434 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:02.349637 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:02.350086 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 22:58:02.350156 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:58:02.350531 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.343332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.14.14.188:39617)
0928 22:58:02.343368 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.343370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.343378 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:58:02.343429 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.343430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.343439 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.343486 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.343738 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.344350 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.345033 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.345231 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.345317 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.347246 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:02.347247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:02.347248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:02.347255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:02.349031 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.349428 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.349430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.349430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.349483 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.349723 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.349724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.349724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.349846 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:02.349904 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:02.350351 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 22:58:02.350447 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:58:02.351557 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:58:02.349872 4284 master_runner.cc:186] Error getting master registration for 127.14.14.190:33313: OK, instance_id { permanent_uuid: "51241efd82074caf8bd53b752838868c" instance_seqno: 1790636281996468 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:58:02.352226 4284 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 22:58:02.348889 4369 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:58:02.352782 4369 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:58:02.352628 4176 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "51241efd82074caf8bd53b752838868c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41720
I20260928 22:58:02.353868 4202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c: Bootstrap starting.
I20260928 22:58:02.354453 4202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c: Neither blocks nor log segments found. Creating new log.
I20260928 22:58:02.354720 4202 log.cc:824] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:58:02.355451 4202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c: No bootstrap required, opened a new log
I20260928 22:58:02.356389 4202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } }
I20260928 22:58:02.356515 4202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51241efd82074caf8bd53b752838868c, State: Initialized, Role: FOLLOWER
I20260928 22:58:02.356621 4202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } }
I20260928 22:58:02.356977 4202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:58:02.356946 4372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } } }
I20260928 22:58:02.357030 4372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:58:02.357120 4176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 22:58:02.357185 4176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:58:02.358047 4176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } }
I20260928 22:58:02.358603 4176 leader_election.cc:288] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [CANDIDATE]: Term 1 election: Requested vote from peers 8f4c4f6bbe274890b015bf87c2836051 (127.14.14.189:42499), b78be04285844e4d82090f93444bbfd7 (127.14.14.188:39617)
I20260928 22:58:02.358762 4326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51241efd82074caf8bd53b752838868c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b78be04285844e4d82090f93444bbfd7"
W20260928 22:58:02.358826 4383 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:58:02.358876 4383 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:58:02.358865 4326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:58:02.359741 4326 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51241efd82074caf8bd53b752838868c in term 1.
I20260928 22:58:02.359647 4134 master_runner.cc:186] Error getting master registration for 127.14.14.189:42499: OK, instance_id { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" instance_seqno: 1790636282151917 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:58:02.362502 4134 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:58:02.364318 4153 leader_election.cc:302] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51241efd82074caf8bd53b752838868c, b78be04285844e4d82090f93444bbfd7; no voters:
I20260928 22:58:02.364563 4372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:58:02.364722 4372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [term 1 LEADER]: Becoming Leader. State: Replica: 51241efd82074caf8bd53b752838868c, State: Running, Role: LEADER
I20260928 22:58:02.364800 4372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } }
I20260928 22:58:02.365161 4372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51241efd82074caf8bd53b752838868c. Latest consensus state: current_term: 1 leader_uuid: "51241efd82074caf8bd53b752838868c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } } }
I20260928 22:58:02.365259 4372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [sys.catalog]: This master's current role is: LEADER
I20260928 22:58:02.365443 4394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:58:02.365694 4394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:58:02.369974 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.361475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.14.14.188:39617)
0928 22:58:02.361513 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.361514 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.361828 (+ 314us) server_negotiation.cc:407] Connection header received
0928 22:58:02.361889 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.361890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.361901 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.361944 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.362658 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.363372 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.364396 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.364680 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.364718 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.366803 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:02.366804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:02.366805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:02.366812 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:02.368451 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.369065 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.369067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.369067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.369120 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.369306 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.369307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.369308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.369447 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:02.369547 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:02.369776 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 22:58:02.369860 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:58:02.372430 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.362614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.14.14.189:42499)
0928 22:58:02.362654 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.362656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.362664 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:58:02.362684 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.362684 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.362692 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.362748 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.363452 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.364147 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.366886 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.367626 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.367692 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.369324 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:02.369326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:02.369326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:02.369334 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:02.371288 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.371618 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.371619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.371620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.371674 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.371911 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.371912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.371912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.372040 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:02.372115 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:02.372250 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 22:58:02.372321 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:58:02.373512 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.364265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.14.14.190:33313)
0928 22:58:02.364441 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.364443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.364450 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:58:02.364469 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.364470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.364478 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.364529 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.366020 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.366765 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.368355 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.368650 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.368709 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.370470 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:02.370472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:02.370472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:02.370479 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:02.372078 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.372544 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.372545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.372545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.372597 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.372917 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.372918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.372934 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.373107 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:02.373202 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:02.373342 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 22:58:02.373418 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260928 22:58:02.378644 4326 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7 [term 1 FOLLOWER]: Refusing update from remote peer 51241efd82074caf8bd53b752838868c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:58:02.379122 4392 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:58:02.380514 4358 sys_catalog.cc:455] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51241efd82074caf8bd53b752838868c. Latest consensus state: current_term: 1 leader_uuid: "51241efd82074caf8bd53b752838868c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } } }
I20260928 22:58:02.380609 4358 sys_catalog.cc:458] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:58:02.381958 4392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51241efd82074caf8bd53b752838868c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } } }
I20260928 22:58:02.382041 4392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [sys.catalog]: This master's current role is: LEADER
I20260928 22:58:02.382563 4358 sys_catalog.cc:455] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51241efd82074caf8bd53b752838868c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } } }
I20260928 22:58:02.382642 4358 sys_catalog.cc:458] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:58:02.382768 4394 catalog_manager.cc:1347] Generated new cluster ID: 8217a3ad0ecc4441b1b79ee8f3a3fa9e
I20260928 22:58:02.382844 4394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:58:02.383001 4395 mvcc.cc:205] Tried to move back new op lower bound from 7334446212619862016 to 7334446212566806528. Current Snapshot: MvccSnapshot[applied={T|T < 7334446212619862016 or (T in {7334446212619862016})}]
I20260928 22:58:02.387248 4277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } attempt: 3
I20260928 22:58:02.390311 4396 mvcc.cc:205] Tried to move back new op lower bound from 7334446212619862016 to 7334446212566806528. Current Snapshot: MvccSnapshot[applied={T|T < 7334446212619862016}]
Sep 28 22:58:02 dist-test-slave-kzfr krb5kdc[4121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636282, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.189@KRBTEST.COM for kudu/127.14.14.188@KRBTEST.COM
I20260928 22:58:02.397038 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.387651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:39617 (local address 127.0.0.1:38172)
0928 22:58:02.387763 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:02.387775 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:02.387813 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:02.388082 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:02.388083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:02.388303 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:02.388352 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.388354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.389935 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.389937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.390786 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:02.390789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.391471 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.391472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:02.391548 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.392142 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:02.392147 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:02.393722 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:02.395880 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.395882 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.395882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.396109 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.396349 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:02.396350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:02.396350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:02.396413 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:02.396699 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:02.396700 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:02.396754 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 22:58:02.396817 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 22:58:02.396931 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 22:58:02.397539 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:02.387885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.14.14.188:39617)
0928 22:58:02.387925 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.387927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.387933 (+ 6us) server_negotiation.cc:407] Connection header received
0928 22:58:02.387973 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.387974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.387988 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.388035 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.388445 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.389194 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.391002 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:02.391335 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:02.391397 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:02.393824 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:02.393825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:02.393826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:02.393833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:02.395805 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.396209 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.396210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.396210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.396294 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:02.396483 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:02.396484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:02.396484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:02.396617 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:02.396675 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:02.396879 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 22:58:02.397003 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:58:02.399922 4277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051: Bootstrap starting.
I20260928 22:58:02.400643 4277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051: Neither blocks nor log segments found. Creating new log.
I20260928 22:58:02.400946 4277 log.cc:824] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051: Log is configured to *not* fsync() on all Append() calls
I20260928 22:58:02.401898 4277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051: No bootstrap required, opened a new log
I20260928 22:58:02.403942 4277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } }
I20260928 22:58:02.404098 4277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4c4f6bbe274890b015bf87c2836051, State: Initialized, Role: FOLLOWER
I20260928 22:58:02.404184 4277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } }
I20260928 22:58:02.404533 4277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:58:02.404592 4250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:58:02.405059 4401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } } }
I20260928 22:58:02.405639 4394 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:58:02.406378 4394 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:58:02.405802 4250 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051 [term 1 FOLLOWER]: Refusing update from remote peer 51241efd82074caf8bd53b752838868c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:58:02.405849 4251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51241efd82074caf8bd53b752838868c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8f4c4f6bbe274890b015bf87c2836051"
I20260928 22:58:02.407272 4392 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:58:02.408026 4251 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51241efd82074caf8bd53b752838868c in term 1.
I20260928 22:58:02.410856 4392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51241efd82074caf8bd53b752838868c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } } }
I20260928 22:58:02.410949 4392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c [sys.catalog]: This master's current role is: LEADER
I20260928 22:58:02.411852 4394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51241efd82074caf8bd53b752838868c: Generated new TSK 0
I20260928 22:58:02.412061 4394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:58:02.412535 4401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:58:02.412837 4401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51241efd82074caf8bd53b752838868c. Latest consensus state: current_term: 1 leader_uuid: "51241efd82074caf8bd53b752838868c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } } }
I20260928 22:58:02.413019 4401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:58:02.413098 4401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51241efd82074caf8bd53b752838868c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51241efd82074caf8bd53b752838868c" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 33313 } } peers { permanent_uuid: "8f4c4f6bbe274890b015bf87c2836051" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 42499 } } peers { permanent_uuid: "b78be04285844e4d82090f93444bbfd7" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 39617 } } }
I20260928 22:58:02.413190 4401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:58:02.415200 4416 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:58:02.415264 4416 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:58:02.424163 4413 mvcc.cc:205] Tried to move back new op lower bound from 7334446212717174784 to 7334446212566806528. Current Snapshot: MvccSnapshot[applied={T|T < 7334446212619862016}]
W20260928 22:58:02.440371 4281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:02.438701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.14.14.189:42499)
0928 22:58:02.438757 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.438759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.439660 (+ 901us) server_negotiation.cc:407] Connection header received
0928 22:58:02.439777 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.439779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.439792 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.439823 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.440302 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51654: BlockingRecv error: recv got EOF from 127.0.0.1:51654 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 22:58:02.441439 4356 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:02.439051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.14.14.188:39617)
0928 22:58:02.439090 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.439092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.440608 (+ 1516us) server_negotiation.cc:407] Connection header received
0928 22:58:02.440883 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.440885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.440895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.440923 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.441377 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38180: BlockingRecv error: recv got EOF from 127.0.0.1:38180 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:58:02.441943 4206 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:02.439413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.14.14.190:33313)
0928 22:58:02.439451 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.439453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.440744 (+ 1291us) server_negotiation.cc:407] Connection header received
0928 22:58:02.440984 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.440986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.440997 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.441025 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.441883 (+ 858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41736: BlockingRecv error: recv got EOF from 127.0.0.1:41736 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 22:58:02.444430 4281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:02.443645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.14.14.189:42499)
0928 22:58:02.443682 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.443684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.443922 (+ 238us) server_negotiation.cc:407] Connection header received
0928 22:58:02.444028 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.444029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.444048 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.444077 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.444382 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51660: BlockingRecv error: recv got EOF from 127.0.0.1:51660 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 22:58:02.447430 4281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:02.446618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.14.14.189:42499)
0928 22:58:02.446652 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.446653 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.446947 (+ 294us) server_negotiation.cc:407] Connection header received
0928 22:58:02.447048 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.447049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.447060 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.447087 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.447387 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51662: BlockingRecv error: recv got EOF from 127.0.0.1:51662 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 22:58:02.451247 4281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:02.450461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.14.14.189:42499)
0928 22:58:02.450494 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.450496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.450748 (+ 252us) server_negotiation.cc:407] Connection header received
0928 22:58:02.450856 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.450857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.450867 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.450897 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.451206 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51678: BlockingRecv error: recv got EOF from 127.0.0.1:51678 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 22:58:02.457083 4281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:02.456281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.14.14.189:42499)
0928 22:58:02.456316 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.456318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.456563 (+ 245us) server_negotiation.cc:407] Connection header received
0928 22:58:02.456662 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.456663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.456674 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.456703 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.457040 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51684: BlockingRecv error: recv got EOF from 127.0.0.1:51684 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 22:58:02.467168 4281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:02.466285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.14.14.189:42499)
0928 22:58:02.466326 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.466328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.466592 (+ 264us) server_negotiation.cc:407] Connection header received
0928 22:58:02.466699 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.466700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.466711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.466754 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.467106 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51698: BlockingRecv error: recv got EOF from 127.0.0.1:51698 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 22:58:02.485471 4281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:02.484490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.14.14.189:42499)
0928 22:58:02.484537 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.484539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.484851 (+ 312us) server_negotiation.cc:407] Connection header received
0928 22:58:02.484958 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.484959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.484972 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.485003 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.485396 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51706: BlockingRecv error: recv got EOF from 127.0.0.1:51706 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 22:58:02.519812 4281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:02.518871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.14.14.189:42499)
0928 22:58:02.518918 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.518921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.519228 (+ 307us) server_negotiation.cc:407] Connection header received
0928 22:58:02.519336 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.519337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.519349 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.519378 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.519754 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51708: BlockingRecv error: recv got EOF from 127.0.0.1:51708 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:58:02.586190 4281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:02.585267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51712 (local address 127.14.14.189:42499)
0928 22:58:02.585328 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.585330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.585466 (+ 136us) server_negotiation.cc:407] Connection header received
0928 22:58:02.585572 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.585574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.585620 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.585689 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.586122 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51712: BlockingRecv error: recv got EOF from 127.0.0.1:51712 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 22:58:02.716382 4281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:02.715427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.14.14.189:42499)
0928 22:58:02.715528 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.715530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.715696 (+ 166us) server_negotiation.cc:407] Connection header received
0928 22:58:02.715803 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.715804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.715819 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.715896 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.716302 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51722: BlockingRecv error: recv got EOF from 127.0.0.1:51722 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 22:58:02.974361 4281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:02.973648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.14.14.189:42499)
0928 22:58:02.973726 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:02.973729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:02.973740 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:58:02.973792 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:02.973793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:02.973807 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:02.973885 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:02.974267 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51724: BlockingRecv error: recv got EOF from 127.0.0.1:51724 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 22:58:03.353340 4369 catalog_manager.cc:1259] Loaded cluster ID: 8217a3ad0ecc4441b1b79ee8f3a3fa9e
I20260928 22:58:03.353421 4369 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7: loading cluster ID for follower catalog manager: success
I20260928 22:58:03.354456 4369 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7: acquiring CA information for follower catalog manager: success
I20260928 22:58:03.354839 4369 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b78be04285844e4d82090f93444bbfd7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:58:03.415747 4416 catalog_manager.cc:1259] Loaded cluster ID: 8217a3ad0ecc4441b1b79ee8f3a3fa9e
I20260928 22:58:03.415880 4416 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051: loading cluster ID for follower catalog manager: success
I20260928 22:58:03.416687 4416 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051: acquiring CA information for follower catalog manager: success
I20260928 22:58:03.417014 4416 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8f4c4f6bbe274890b015bf87c2836051: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:58:03.490041 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:03.487565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50532 (local address 127.14.14.189:42499)
0928 22:58:03.487662 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:03.487664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:03.487675 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:58:03.487706 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:03.487707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:03.487756 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:03.487843 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:58:03.487997 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:03.488508 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:03.489249 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:03.489454 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:03.489508 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:03.489684 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:58:03.489801 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:58:03.489862 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:03.489899 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 22:58:03.489918 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":50,"threads_started":1}
I20260928 22:58:03.494181 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:03.491863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.14.14.188:39617)
0928 22:58:03.491983 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:03.491984 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:03.491992 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:58:03.492045 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:03.492046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:03.492094 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:03.492178 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:58:03.492346 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:03.492811 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:03.493492 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:03.493747 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:03.493798 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:03.493922 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:58:03.494014 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:58:03.494068 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:03.494106 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 22:58:03.494122 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
I20260928 22:58:03.498968 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:03.495867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.14.14.190:33313)
0928 22:58:03.495961 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:03.495963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:03.495972 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:58:03.496047 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:03.496048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:03.496095 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:03.496193 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:58:03.496367 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:03.496976 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:03.498055 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:03.498357 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:03.498429 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:03.498622 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:58:03.498734 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:58:03.498816 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:03.498861 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 22:58:03.498887 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":46,"threads_started":1}
I20260928 22:58:03.498963 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:03.495944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.14.14.189:42499)
0928 22:58:03.495983 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:03.495985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:03.496164 (+ 179us) server_negotiation.cc:407] Connection header received
0928 22:58:03.496197 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:03.496198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:03.496207 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:03.496257 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:58:03.496430 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:03.497159 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:03.498142 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:03.498437 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:03.498498 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:03.498670 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:58:03.498710 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:58:03.498759 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:03.498873 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 22:58:03.498895 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:58:03.499733 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:03.496255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.14.14.188:39617)
0928 22:58:03.496294 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:03.496296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:03.496384 (+ 88us) server_negotiation.cc:407] Connection header received
0928 22:58:03.496461 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:03.496463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:03.496471 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:03.496520 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:58:03.496643 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:03.497328 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:03.498909 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:03.499212 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:03.499249 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:03.499477 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:58:03.499510 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:58:03.499537 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:03.499626 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 22:58:03.499645 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:58:03.500816 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 4134
I20260928 22:58:03.506115 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 4209
I20260928 22:58:03.510887 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 4284
2026-09-28T22:58:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2777 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:58:03 dist-test-slave-kzfr krb5kdc[4502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:58:03 dist-test-slave-kzfr krb5kdc[4502](info): set up 2 sockets
Sep 28 22:58:03 dist-test-slave-kzfr krb5kdc[4502](info): commencing operation
krb5kdc: starting...
W20260928 22:58:04.554231 14394 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/test-user.keytab.
Sep 28 22:58:04 dist-test-slave-kzfr krb5kdc[4502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:58:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.14.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
Entry for principal kudu/127.14.14.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab.
I20260928 22:58:04.677057 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44953
--webserver_interface=127.14.14.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:44953,127.14.14.189:41309,127.14.14.188:41181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:58:04.757400 4515 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:58:04.757674 4515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:58:04.757707 4515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:58:04.757750 4515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:58:04.758894 4515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:58:04.758968 4515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:58:04.758996 4515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:58:04.759027 4515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:58:04.759053 4515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:58:04.760550 4515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:44953,127.14.14.189:41309,127.14.14.188:41181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.190.keytab
--principal=kudu/127.14.14.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.190:44953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.14.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:58:04.760842 4515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:58:04.761076 4515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:58:04.766028 4521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:58:04.766038 4520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:58:04.766108 4523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:58:04.766146 4515 server_base.cc:1034] running on GCE node
I20260928 22:58:04.766505 4515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:58:04.766762 4515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:58:04.767902 4515 hybrid_clock.cc:648] HybridClock initialized: now 1790636284767888 us; error 26 us; skew 500 ppm
Sep 28 22:58:04 dist-test-slave-kzfr krb5kdc[4502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636284, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:58:04.779492 4515 init.cc:377] Logged in from keytab as kudu/127.14.14.190@KRBTEST.COM (short username kudu)
I20260928 22:58:04.779986 4515 webserver.cc:460] Webserver started at http://127.14.14.190:41765/ using document root <none> and password file <none>
I20260928 22:58:04.780244 4515 fs_manager.cc:359] Metadata directory not provided
I20260928 22:58:04.780316 4515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:58:04.780469 4515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:58:04.781730 4515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/data/instance:
uuid: "a72a333765aa4cd9a006339fee99bffa"
format_stamp: "Formatted at 2026-09-28 22:58:04 on dist-test-slave-kzfr"
server_key: "62a72554bf173681f83f2d20ce82cd75"
server_key_iv: "ade025d2e49dad68d0c9d8a8b3bea7cc"
server_key_version: "encryptionkey@0"
I20260928 22:58:04.782101 4515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/wal/instance:
uuid: "a72a333765aa4cd9a006339fee99bffa"
format_stamp: "Formatted at 2026-09-28 22:58:04 on dist-test-slave-kzfr"
server_key: "62a72554bf173681f83f2d20ce82cd75"
server_key_iv: "ade025d2e49dad68d0c9d8a8b3bea7cc"
server_key_version: "encryptionkey@0"
I20260928 22:58:04.783475 4515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:58:04.784240 4530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:58:04.784458 4515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:58:04.784552 4515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/wal
uuid: "a72a333765aa4cd9a006339fee99bffa"
format_stamp: "Formatted at 2026-09-28 22:58:04 on dist-test-slave-kzfr"
server_key: "62a72554bf173681f83f2d20ce82cd75"
server_key_iv: "ade025d2e49dad68d0c9d8a8b3bea7cc"
server_key_version: "encryptionkey@0"
I20260928 22:58:04.784636 4515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:58:04.799516 4515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:58:04.802084 4515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:58:04.802256 4515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:58:04.806246 4515 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.190:44953
I20260928 22:58:04.806288 4582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.190:44953 every 8 connection(s)
I20260928 22:58:04.806656 4515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/data/info.pb
I20260928 22:58:04.807840 4583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:58:04.809393 4583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:58:04.812011 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 4515
I20260928 22:58:04.812096 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-0/wal/instance
I20260928 22:58:04.812326 14394 external_mini_cluster.cc:1428] Setting key 488d0f7e953d1cabd215070ae4a8e75f
Sep 28 22:58:04 dist-test-slave-kzfr krb5kdc[4502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636284, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:58:04.819633 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:04.810158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44953 (local address 127.0.0.1:46562)
0928 22:58:04.810340 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:04.810379 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:04.810829 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:04.811066 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:04.811068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:04.811307 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:04.811461 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:04.811465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:04.812658 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:04.812658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:04.813341 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:04.813343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:04.813971 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:04.813972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:04.814034 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:04.814595 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:04.814604 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:04.815966 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:04.818131 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:04.818137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:04.818139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:04.818359 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:04.818638 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:04.818639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:04.818639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:04.818718 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:04.819173 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:04.819178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:04.819266 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 22:58:04.819344 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:58:04.819469 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":117,"threads_started":1}
I20260928 22:58:04.819728 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:04.810171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.14.14.190:44953)
0928 22:58:04.810356 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:04.810366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:04.810474 (+ 108us) server_negotiation.cc:407] Connection header received
0928 22:58:04.810953 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:04.810955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:04.810972 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:04.811061 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:04.811551 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:04.812550 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:04.813442 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:04.813854 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:04.813983 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:04.816053 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:04.816062 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:04.816062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:04.816070 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:04.818049 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:04.818446 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:04.818450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:04.818451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:04.818563 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:04.818794 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:04.818794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:04.818795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:04.819097 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:04.819193 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:04.819418 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 22:58:04.819557 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":146,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.14.14.189@KRBTEST.COM; defaulting to no policy
I20260928 22:58:04.820709 4583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:58:04.821131 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:04.820951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:41309 (local address 127.0.0.1:40960)
0928 22:58:04.821026 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:04.821100 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:58:04.821288 4533 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.14.189:41309: connect: Connection refused (error 111)
I20260928 22:58:04.821754 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:04.821666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:41309 (local address 127.0.0.1:40962)
0928 22:58:04.821711 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:04.821738 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:58:04.821921 4583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.189:41309: Network error: Client connection negotiation failed: client connection to 127.14.14.189:41309: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.189.keytab.
Entry for principal kudu/127.14.14.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.189.keytab.
I20260928 22:58:04.839110 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.189:41309
--webserver_interface=127.14.14.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.189.keytab
--principal=kudu/127.14.14.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:44953,127.14.14.189:41309,127.14.14.188:41181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:58:04.871145 4583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } attempt: 1
I20260928 22:58:04.871647 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:04.871506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:41309 (local address 127.0.0.1:40972)
0928 22:58:04.871569 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:04.871618 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 22:58:04.872004 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:04.871905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:41309 (local address 127.0.0.1:40986)
0928 22:58:04.871951 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:04.871976 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 22:58:04.872097 4583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.189:41309: Network error: Client connection negotiation failed: client connection to 127.14.14.189:41309: connect: Connection refused (error 111)
W20260928 22:58:04.920439 4590 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:58:04.920688 4590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:58:04.920718 4590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:58:04.920759 4590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:58:04.921967 4590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:58:04.922034 4590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:58:04.922060 4590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:58:04.922091 4590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:58:04.922118 4590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:58:04.923609 4590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:44953,127.14.14.189:41309,127.14.14.188:41181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.189.keytab
--principal=kudu/127.14.14.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.189:41309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.14.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:58:04.923884 4590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:58:04.924126 4590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:58:04.929044 4596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:58:04.929059 4598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:58:04.929257 4590 server_base.cc:1034] running on GCE node
W20260928 22:58:04.929019 4595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:58:04.929509 4590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:58:04.929826 4590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:58:04.930974 4590 hybrid_clock.cc:648] HybridClock initialized: now 1790636284930962 us; error 28 us; skew 500 ppm
Sep 28 22:58:04 dist-test-slave-kzfr krb5kdc[4502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636284, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:58:04.933435 4590 init.cc:377] Logged in from keytab as kudu/127.14.14.189@KRBTEST.COM (short username kudu)
I20260928 22:58:04.933897 4590 webserver.cc:460] Webserver started at http://127.14.14.189:40667/ using document root <none> and password file <none>
I20260928 22:58:04.934144 4590 fs_manager.cc:359] Metadata directory not provided
I20260928 22:58:04.934190 4590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:58:04.934279 4590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:58:04.935247 4590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/data/instance:
uuid: "ad763a8b38024bc08eb1764247791802"
format_stamp: "Formatted at 2026-09-28 22:58:04 on dist-test-slave-kzfr"
server_key: "39085d17c2a16e1010e49cc6a60e5c76"
server_key_iv: "f73c40f1afbffee886629f43c07d0bc2"
server_key_version: "encryptionkey@0"
I20260928 22:58:04.935544 4590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/wal/instance:
uuid: "ad763a8b38024bc08eb1764247791802"
format_stamp: "Formatted at 2026-09-28 22:58:04 on dist-test-slave-kzfr"
server_key: "39085d17c2a16e1010e49cc6a60e5c76"
server_key_iv: "f73c40f1afbffee886629f43c07d0bc2"
server_key_version: "encryptionkey@0"
I20260928 22:58:04.936801 4590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:58:04.937489 4605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:58:04.937703 4590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:58:04.937819 4590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/wal
uuid: "ad763a8b38024bc08eb1764247791802"
format_stamp: "Formatted at 2026-09-28 22:58:04 on dist-test-slave-kzfr"
server_key: "39085d17c2a16e1010e49cc6a60e5c76"
server_key_iv: "f73c40f1afbffee886629f43c07d0bc2"
server_key_version: "encryptionkey@0"
I20260928 22:58:04.937899 4590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:58:04.940239 4583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } attempt: 2
I20260928 22:58:04.940644 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:04.940496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:41309 (local address 127.0.0.1:40990)
0928 22:58:04.940564 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:04.940604 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 22:58:04.941015 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:04.940916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:41309 (local address 127.0.0.1:40998)
0928 22:58:04.940963 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:04.940991 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.189:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 22:58:04.941113 4583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.189:41309: Network error: Client connection negotiation failed: client connection to 127.14.14.189:41309: connect: Connection refused (error 111)
I20260928 22:58:04.954576 4590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:58:04.956951 4590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:58:04.957122 4590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:58:04.961067 4590 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.189:41309
I20260928 22:58:04.961143 4657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.189:41309 every 8 connection(s)
I20260928 22:58:04.961469 4590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/data/info.pb
I20260928 22:58:04.962546 4658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:58:04.964138 4658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:58:04.964553 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 4590
I20260928 22:58:04.964689 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-1/wal/instance
I20260928 22:58:04.964918 14394 external_mini_cluster.cc:1428] Setting key 1322773de88b443a3aceb6ec8c24765c
Sep 28 22:58:04 dist-test-slave-kzfr krb5kdc[4502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636284, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.189@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.14.188@KRBTEST.COM; defaulting to no policy
I20260928 22:58:04.974934 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:04.964936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.14.14.190:44953)
0928 22:58:04.965009 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:04.965012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:04.965291 (+ 279us) server_negotiation.cc:407] Connection header received
0928 22:58:04.966074 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:04.966076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:04.966088 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:04.966131 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:04.966907 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:04.967647 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:04.969206 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:04.969429 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:04.969499 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:04.972044 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:04.972046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:04.972047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:04.972056 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:04.973668 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:04.974061 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:04.974062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:04.974063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:04.974115 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:04.974336 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:04.974336 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:04.974336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:04.974438 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:04.974488 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:04.974749 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 22:58:04.974839 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:58:04.975136 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:04.964909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44953 (local address 127.0.0.1:46564)
0928 22:58:04.965134 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:04.965190 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:04.965960 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:04.966203 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:04.966207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:04.966592 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:04.966813 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:04.966818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:04.967762 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:04.967763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:04.969128 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:04.969131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:04.969547 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:04.969549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:04.969765 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:04.970292 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:04.970303 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:04.971946 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:04.973762 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:04.973768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:04.973770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:04.973993 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:04.974175 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:04.974176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:04.974176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:04.974280 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:04.974491 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:04.974496 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:04.974611 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 22:58:04.974722 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 22:58:04.974895 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":119,"threads_started":1}
I20260928 22:58:04.975765 4658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.14.188@KRBTEST.COM" created.
Sep 28 22:58:04 dist-test-slave-kzfr krb5kdc[4502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636284, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.189@KRBTEST.COM for kudu/127.14.14.189@KRBTEST.COM
I20260928 22:58:04.983038 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:04.976011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:41309 (local address 127.0.0.1:41006)
0928 22:58:04.976062 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:04.976073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:04.976118 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:04.976360 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:04.976361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:04.976503 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:04.976551 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:04.976552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:04.977406 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:04.977407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:04.978162 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:04.978164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:04.978628 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:04.978629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:04.978711 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:04.979105 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:04.979110 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:04.980246 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:04.981996 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:04.981997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:04.981997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:04.982157 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:04.982378 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:04.982379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:04.982379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:04.982434 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:04.982716 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:04.982717 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:04.982752 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 22:58:04.982825 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:58:04.982924 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:58:04.983103 4662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:04.976076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41006 (local address 127.14.14.189:41309)
0928 22:58:04.976232 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:04.976235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:04.976250 (+ 15us) server_negotiation.cc:407] Connection header received
0928 22:58:04.976266 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:04.976268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:04.976279 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:04.976361 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:04.976627 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:04.977303 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:04.978250 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:04.978546 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:04.978605 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:04.980329 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:04.980337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:04.980337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:04.980351 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:04.981903 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:04.982249 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:04.982252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:04.982253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:04.982310 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:04.982505 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:04.982506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:04.982506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:04.982644 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:04.982737 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:04.982856 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 22:58:04.982977 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260928 22:58:04.983870 4658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:58:04.984205 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:04.984107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:41181 (local address 127.0.0.1:33428)
0928 22:58:04.984140 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:04.984183 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 22:58:04.984380 4606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
I20260928 22:58:04.984850 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:04.984701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:41181 (local address 127.0.0.1:33436)
0928 22:58:04.984776 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:04.984819 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 22:58:04.984987 4658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.188:41181: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.14.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.188.keytab.
Entry for principal kudu/127.14.14.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.188.keytab.
I20260928 22:58:04.993498 14394 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdZKSl/build/release/bin/kudu
/tmp/dist-test-taskkdZKSl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.188:41181
--webserver_interface=127.14.14.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.188.keytab
--principal=kudu/127.14.14.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.14.148:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.14.190:44953,127.14.14.189:41309,127.14.14.188:41181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:58:05.032261 4583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } attempt: 3
I20260928 22:58:05.034232 4658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } attempt: 1
I20260928 22:58:05.034657 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.034520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:41181 (local address 127.0.0.1:33452)
0928 22:58:05.034595 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:05.034628 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 22:58:05.035012 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.034915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:41181 (local address 127.0.0.1:33456)
0928 22:58:05.034961 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:05.034993 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:58:05.035105 4658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.188:41181: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
Sep 28 22:58:05 dist-test-slave-kzfr krb5kdc[4502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636284, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for kudu/127.14.14.189@KRBTEST.COM
I20260928 22:58:05.040596 4662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.032733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.14.14.189:41309)
0928 22:58:05.032803 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.032807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.032887 (+ 80us) server_negotiation.cc:407] Connection header received
0928 22:58:05.032994 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.032995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.033008 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.033063 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.033445 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.034052 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.035182 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.035521 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.035590 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.037822 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:05.037823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:05.037823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:05.037829 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:05.039408 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.039800 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.039801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.039801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.039853 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.040054 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.040055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.040055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.040197 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:05.040261 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:05.040463 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 22:58:05.040535 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:58:05.040623 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.032729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:41309 (local address 127.0.0.1:41018)
0928 22:58:05.032806 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:05.032820 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:05.032896 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:05.033092 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:05.033092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:05.033284 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:05.033361 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:05.033363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.034240 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.034242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:05.035081 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:05.035084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.035599 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.035600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:05.035704 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.036226 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:05.036231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:05.037718 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:05.039493 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:05.039494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:05.039494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:05.039723 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:05.039921 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:05.039922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:05.039922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:05.039992 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:05.040270 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:05.040271 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:05.040320 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 22:58:05.040418 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 22:58:05.040522 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 22:58:05.041438 4583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:58:05.041836 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.041722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:41181 (local address 127.0.0.1:33470)
0928 22:58:05.041773 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:05.041808 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:58:05.042232 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.042143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:41181 (local address 127.0.0.1:33474)
0928 22:58:05.042187 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:05.042213 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:58:05.042332 4583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.188:41181: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
I20260928 22:58:05.073503 4583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } attempt: 1
I20260928 22:58:05.074055 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.073892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:41181 (local address 127.0.0.1:33478)
0928 22:58:05.073977 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:05.074023 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 22:58:05.074517 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.074421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:41181 (local address 127.0.0.1:33480)
0928 22:58:05.074467 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:05.074494 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:58:05.074644 4583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.188:41181: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
W20260928 22:58:05.077346 4665 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:58:05.077577 4665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:58:05.077636 4665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:58:05.077677 4665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:58:05.078824 4665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:58:05.078889 4665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:58:05.078915 4665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:58:05.078946 4665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:58:05.078974 4665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:58:05.080442 4665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.14.148:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.14.190:44953,127.14.14.189:41309,127.14.14.188:41181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/krb5kdc/kudu_127.14.14.188.keytab
--principal=kudu/127.14.14.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.14.188:41181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.14.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:58:05.080718 4665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:58:05.080950 4665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:58:05.086009 4665 server_base.cc:1034] running on GCE node
W20260928 22:58:05.085984 4670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:58:05.085997 4671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:58:05.085992 4673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:58:05.086428 4665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:58:05.086776 4665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:58:05.087942 4665 hybrid_clock.cc:648] HybridClock initialized: now 1790636285087927 us; error 25 us; skew 500 ppm
Sep 28 22:58:05 dist-test-slave-kzfr krb5kdc[4502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636285, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:58:05.090448 4665 init.cc:377] Logged in from keytab as kudu/127.14.14.188@KRBTEST.COM (short username kudu)
I20260928 22:58:05.090915 4665 webserver.cc:460] Webserver started at http://127.14.14.188:33991/ using document root <none> and password file <none>
I20260928 22:58:05.091150 4665 fs_manager.cc:359] Metadata directory not provided
I20260928 22:58:05.091220 4665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:58:05.091323 4665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:58:05.092337 4665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/data/instance:
uuid: "6facde8029b5496d8fc6b8acd6e791b0"
format_stamp: "Formatted at 2026-09-28 22:58:05 on dist-test-slave-kzfr"
server_key: "e33c47918a9ddfdcb6cedbd1b7ca678e"
server_key_iv: "b284410a10f6ed36827bd4ce05640766"
server_key_version: "encryptionkey@0"
I20260928 22:58:05.092633 4665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/wal/instance:
uuid: "6facde8029b5496d8fc6b8acd6e791b0"
format_stamp: "Formatted at 2026-09-28 22:58:05 on dist-test-slave-kzfr"
server_key: "e33c47918a9ddfdcb6cedbd1b7ca678e"
server_key_iv: "b284410a10f6ed36827bd4ce05640766"
server_key_version: "encryptionkey@0"
I20260928 22:58:05.093924 4665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:58:05.094650 4680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:58:05.094848 4665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:58:05.094937 4665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/data,/tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/wal
uuid: "6facde8029b5496d8fc6b8acd6e791b0"
format_stamp: "Formatted at 2026-09-28 22:58:05 on dist-test-slave-kzfr"
server_key: "e33c47918a9ddfdcb6cedbd1b7ca678e"
server_key_iv: "b284410a10f6ed36827bd4ce05640766"
server_key_version: "encryptionkey@0"
I20260928 22:58:05.094990 4665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:58:05.103255 4658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } attempt: 2
I20260928 22:58:05.103720 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.103578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:41181 (local address 127.0.0.1:33488)
0928 22:58:05.103642 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:05.103682 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 22:58:05.104070 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.103993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:41181 (local address 127.0.0.1:33498)
0928 22:58:05.104033 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:05.104055 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 22:58:05.104187 4658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.14.188:41181: Network error: Client connection negotiation failed: client connection to 127.14.14.188:41181: connect: Connection refused (error 111)
I20260928 22:58:05.108935 4665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:58:05.111316 4665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:58:05.111462 4665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:58:05.115336 4665 rpc_server.cc:307] RPC server started. Bound to: 127.14.14.188:41181
I20260928 22:58:05.115389 4732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.14.188:41181 every 8 connection(s)
I20260928 22:58:05.115753 4665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/data/info.pb
I20260928 22:58:05.116750 4733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:58:05.118145 14394 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdZKSl/build/release/bin/kudu as pid 4665
I20260928 22:58:05.118265 14394 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdZKSl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790636016256058-14394-0/minicluster-data/master-2/wal/instance
I20260928 22:58:05.118459 14394 external_mini_cluster.cc:1428] Setting key c9166dbba0b7f5f69ce4f1fb9de04da4
I20260928 22:58:05.118368 4733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:58:05 dist-test-slave-kzfr krb5kdc[4502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636285, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.188@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
Sep 28 22:58:05 dist-test-slave-kzfr krb5kdc[4502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.188@KRBTEST.COM
I20260928 22:58:05.127897 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.119104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.14.14.190:44953)
0928 22:58:05.119178 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.119180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.119338 (+ 158us) server_negotiation.cc:407] Connection header received
0928 22:58:05.119812 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.119814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.119826 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.119885 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.120441 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.120926 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.121859 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.122063 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.122147 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.124486 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:05.124488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:05.124488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:05.124495 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:05.126608 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.127021 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.127023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.127023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.127079 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.127326 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.127327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.127327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.127441 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:05.127515 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:05.127704 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 22:58:05.127790 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:58:05.127960 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.118967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33512 (local address 127.14.14.188:41181)
0928 22:58:05.119269 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.119273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.119306 (+ 33us) server_negotiation.cc:407] Connection header received
0928 22:58:05.119705 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.119711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.119730 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.119840 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.120468 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.121233 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.122064 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.122355 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.122431 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.124917 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:05.124924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:05.124925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:05.124931 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:05.126521 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.126888 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.126891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.126892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.126981 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.127189 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.127190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.127190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.127394 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:05.127532 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:05.127624 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 22:58:05.127770 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":75,"threads_started":1}
I20260928 22:58:05.128079 4737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.119048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.190:44953 (local address 127.0.0.1:46578)
0928 22:58:05.119221 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:05.119264 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:05.119714 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:05.119905 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:05.119909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:05.120168 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:05.120349 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:05.120353 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.121009 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.121010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:05.121776 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:05.121778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.122177 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.122178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:05.122297 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.122772 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:05.122781 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:05.124389 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:05.126711 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:05.126717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:05.126719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:05.126931 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:05.127152 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:05.127153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:05.127153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:05.127247 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:05.127506 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:05.127512 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:05.127576 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 22:58:05.127678 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 22:58:05.127939 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":132,"thread_start_us":83,"threads_started":1}
I20260928 22:58:05.128562 4733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:58:05 dist-test-slave-kzfr krb5kdc[4502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636285, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.188@KRBTEST.COM for kudu/127.14.14.189@KRBTEST.COM
Sep 28 22:58:05 dist-test-slave-kzfr krb5kdc[4502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.190@KRBTEST.COM
I20260928 22:58:05.135679 4737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.128829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.189:41309 (local address 127.0.0.1:41024)
0928 22:58:05.128864 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:05.128875 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:05.128912 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:05.129077 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:05.129077 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:05.129181 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:05.129215 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:05.129216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.130029 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.130030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:05.130535 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:05.130537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.131013 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.131014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:05.131079 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.131586 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:05.131591 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:05.132886 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:05.134616 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:05.134617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:05.134618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:05.134795 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:05.135047 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:05.135048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:05.135048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:05.135100 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:05.135385 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:05.135386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:05.135418 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 22:58:05.135500 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 22:58:05.135587 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 22:58:05.135713 4662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.128927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.14.14.189:41309)
0928 22:58:05.128973 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.128975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.128982 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:58:05.129006 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.129007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.129017 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.129065 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.129286 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.129958 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.130613 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.130849 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.130920 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.132972 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:05.132974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:05.132974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:05.132982 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:05.134514 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.134867 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.134868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.134869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.134971 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.135170 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.135171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.135171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.135316 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:05.135389 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:05.135545 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 22:58:05.135619 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:58:05.136518 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.128806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46590 (local address 127.14.14.190:44953)
0928 22:58:05.128839 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.128840 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.128867 (+ 27us) server_negotiation.cc:407] Connection header received
0928 22:58:05.128888 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.128888 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.128898 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.128943 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.129498 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.130031 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.130691 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.130863 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.131038 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.133812 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:05.133813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:05.133814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:05.133821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:05.135387 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.135689 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.135690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.135690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.135743 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.135962 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.135963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.135963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.136069 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:05.136192 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:05.136313 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 22:58:05.136391 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 22:58:05.136529 4733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:58:05 dist-test-slave-kzfr krb5kdc[4502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636285, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.188@KRBTEST.COM for kudu/127.14.14.188@KRBTEST.COM
Sep 28 22:58:05 dist-test-slave-kzfr krb5kdc[4502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.14.189@KRBTEST.COM
I20260928 22:58:05.143519 4737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.136824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:41181 (local address 127.0.0.1:33520)
0928 22:58:05.136865 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:05.136874 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:05.136906 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:05.137067 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:05.137068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:05.137212 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:05.137246 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:05.137248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.138006 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.138007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:05.138507 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:05.138508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.138835 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.138836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:05.138888 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.139332 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:05.139336 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:05.140725 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:05.142581 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:05.142582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:05.142583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:05.142738 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:05.142951 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:05.142952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:05.142952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:05.143020 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:05.143303 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:05.143303 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:05.143335 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 22:58:05.143386 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:58:05.143459 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 22:58:05.143873 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.136860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33520 (local address 127.14.14.188:41181)
0928 22:58:05.136956 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.136958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.136972 (+ 14us) server_negotiation.cc:407] Connection header received
0928 22:58:05.136987 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.136987 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.136994 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.137066 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.137318 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.137931 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.138581 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.138767 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.138817 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.140814 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:05.140815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:05.140815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:05.140821 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:05.142501 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.142818 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.142819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.142820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.142881 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.143096 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.143097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.143098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.143232 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:05.143307 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:05.143411 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 22:58:05.143518 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":79}
I20260928 22:58:05.145061 4662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.137155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41040 (local address 127.14.14.189:41309)
0928 22:58:05.137192 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.137194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.137251 (+ 57us) server_negotiation.cc:407] Connection header received
0928 22:58:05.137330 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.137331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.137341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.137399 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.137888 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.138369 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.139169 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.139445 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.139501 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.142107 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:05.142109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:05.142109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:05.142116 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:05.143749 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.144254 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.144255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.144256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.144334 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.144538 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.144540 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.144540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.144658 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:05.144832 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:05.144869 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 22:58:05.144941 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:58:05.145375 4733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0: Bootstrap starting.
I20260928 22:58:05.146497 4733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0: Neither blocks nor log segments found. Creating new log.
I20260928 22:58:05.146728 14394 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:58:05.146844 4733 log.cc:824] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0: Log is configured to *not* fsync() on all Append() calls
I20260928 22:58:05.147673 4733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0: No bootstrap required, opened a new log
I20260928 22:58:05.147575 4557 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a72a333765aa4cd9a006339fee99bffa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46590
I20260928 22:58:05.149693 4733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } }
I20260928 22:58:05.149780 4583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } attempt: 2
I20260928 22:58:05.149845 4733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6facde8029b5496d8fc6b8acd6e791b0, State: Initialized, Role: FOLLOWER
I20260928 22:58:05.149927 4733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } }
I20260928 22:58:05.150260 4733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:58:05.151641 4739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } } }
I20260928 22:58:05.152709 4739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 22:58:05 dist-test-slave-kzfr krb5kdc[4502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636284, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.190@KRBTEST.COM for kudu/127.14.14.188@KRBTEST.COM
I20260928 22:58:05.153559 4665 master_runner.cc:186] Error getting master registration for 127.14.14.190:44953: OK, instance_id { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" instance_seqno: 1790636284805056 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:58:05.157016 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:58:05.157682 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.150109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33526 (local address 127.14.14.188:41181)
0928 22:58:05.150249 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.150250 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.150257 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:58:05.150273 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.150274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.150282 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.150357 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.150670 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.151376 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.152466 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.152780 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.152853 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.154911 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:05.154913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:05.154913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:05.154920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:05.156280 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.156685 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.156686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.156686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.156749 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.156971 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.157013 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.157014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.157219 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:05.157303 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:05.157482 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 22:58:05.157561 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260928 22:58:05.157704 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.150058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:41181 (local address 127.0.0.1:33526)
0928 22:58:05.150118 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:05.150129 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:05.150170 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:05.150367 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:05.150368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:05.150539 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:05.150584 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:05.150586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.151496 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.151497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:05.152358 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:05.152361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.152909 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.152910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:05.152975 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.153406 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:05.153410 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:05.154811 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:05.156390 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:05.156391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:05.156392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:05.156592 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:05.156822 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:05.156823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:05.156824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:05.156893 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:05.157308 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:05.157309 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:05.157360 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 22:58:05.157457 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 22:58:05.157562 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 22:58:05.158970 4583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa: Bootstrap starting.
I20260928 22:58:05.159628 4583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa: Neither blocks nor log segments found. Creating new log.
I20260928 22:58:05.159924 4583 log.cc:824] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa: Log is configured to *not* fsync() on all Append() calls
I20260928 22:58:05.160689 4583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa: No bootstrap required, opened a new log
W20260928 22:58:05.152182 4750 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:58:05.165650 4750 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:58:05.172073 4583 raft_consensus.cc:348] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } }
I20260928 22:58:05.172264 4583 raft_consensus.cc:729] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72a333765aa4cd9a006339fee99bffa, State: Initialized, Role: FOLLOWER
I20260928 22:58:05.172355 4583 consensus_queue.cc:260] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } }
I20260928 22:58:05.172717 4583 sys_catalog.cc:564] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:58:05.172780 4557 raft_consensus.cc:482] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 22:58:05.172856 4557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:58:05.173841 4557 raft_consensus.cc:504] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } }
I20260928 22:58:05.174111 4557 leader_election.cc:288] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [CANDIDATE]: Term 1 election: Requested vote from peers ad763a8b38024bc08eb1764247791802 (127.14.14.189:41309), 6facde8029b5496d8fc6b8acd6e791b0 (127.14.14.188:41181)
W20260928 22:58:05.175175 4764 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:58:05.175235 4764 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:58:05.172724 4753 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } } }
I20260928 22:58:05.175990 4753 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:58:05.175712 4515 master_runner.cc:186] Error getting master registration for 127.14.14.189:41309: OK, instance_id { permanent_uuid: "ad763a8b38024bc08eb1764247791802" instance_seqno: 1790636284959981 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:58:05.178121 4515 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:58:05.178272 4707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a72a333765aa4cd9a006339fee99bffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6facde8029b5496d8fc6b8acd6e791b0"
I20260928 22:58:05.178369 4707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:58:05.179351 4707 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72a333765aa4cd9a006339fee99bffa in term 1.
I20260928 22:58:05.179613 4531 leader_election.cc:302] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6facde8029b5496d8fc6b8acd6e791b0, a72a333765aa4cd9a006339fee99bffa; no voters:
I20260928 22:58:05.180275 4753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:58:05.181077 4753 raft_consensus.cc:686] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [term 1 LEADER]: Becoming Leader. State: Replica: a72a333765aa4cd9a006339fee99bffa, State: Running, Role: LEADER
I20260928 22:58:05.181298 4753 consensus_queue.cc:237] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } }
I20260928 22:58:05.182606 4753 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader a72a333765aa4cd9a006339fee99bffa. Latest consensus state: current_term: 1 leader_uuid: "a72a333765aa4cd9a006339fee99bffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } } }
I20260928 22:58:05.182829 4753 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [sys.catalog]: This master's current role is: LEADER
I20260928 22:58:05.185899 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.175622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46606 (local address 127.14.14.190:44953)
0928 22:58:05.175668 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.175671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.176692 (+ 1021us) server_negotiation.cc:407] Connection header received
0928 22:58:05.176817 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.176818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.176829 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.176903 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.177741 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.178481 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.179827 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.180136 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.180208 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.182063 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:05.182064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:05.182065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:05.182073 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:05.184650 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.185054 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.185056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.185056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.185124 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.185329 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.185330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.185330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.185476 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:05.185628 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:05.185719 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 22:58:05.185802 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:58:05.194150 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.181819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.14.14.188:41181)
0928 22:58:05.181880 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.181882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.185973 (+ 4091us) server_negotiation.cc:407] Connection header received
0928 22:58:05.185999 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.186000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.186009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.186052 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.186935 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.187673 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.188715 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.189016 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.189075 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.190600 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:05.190601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:05.190602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:05.190609 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:05.192451 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.193071 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.193072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.193073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.193139 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.193398 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.193399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.193399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.193524 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:05.193643 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:05.193962 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 22:58:05.194053 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:58:05.195313 4658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } attempt: 3
I20260928 22:58:05.195534 4775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:58:05.195745 4775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:58:05.197827 4707 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0 [term 1 FOLLOWER]: Refusing update from remote peer a72a333765aa4cd9a006339fee99bffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:58:05.198201 4753 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:58:05.199869 4662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.176357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41050 (local address 127.14.14.189:41309)
0928 22:58:05.176437 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.176440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.182147 (+ 5707us) server_negotiation.cc:407] Connection header received
0928 22:58:05.182260 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.182262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.182277 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.182311 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.183018 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.183807 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.190704 (+ 6897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.191051 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.191099 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.192969 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:05.192971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:05.192971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:05.192981 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:05.194867 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.195512 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.195513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.195513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.195587 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.199213 (+ 3626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.199216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.199216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.199360 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:05.199434 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:05.199665 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 22:58:05.199773 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 22:58:05.199824 4739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a72a333765aa4cd9a006339fee99bffa. Latest consensus state: current_term: 1 leader_uuid: "a72a333765aa4cd9a006339fee99bffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } } }
I20260928 22:58:05.199914 4739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 22:58:05 dist-test-slave-kzfr krb5kdc[4502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636284, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.14.189@KRBTEST.COM for kudu/127.14.14.188@KRBTEST.COM
I20260928 22:58:05.203751 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.195807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33542 (local address 127.14.14.188:41181)
0928 22:58:05.195844 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.195846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.195853 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:58:05.195872 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.195872 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.195881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.196247 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.196279 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.197221 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.198303 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.198577 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.198641 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.200895 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:58:05.200897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:58:05.200897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:58:05.200924 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:58:05.202587 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.202981 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.202982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.202982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.203038 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:58:05.203238 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:58:05.203238 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:58:05.203239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:58:05.203364 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:58:05.203436 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:05.203589 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 22:58:05.203662 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:58:05.203752 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:05.195587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.14.188:41181 (local address 127.0.0.1:33542)
0928 22:58:05.195684 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 22:58:05.195695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:58:05.195732 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:58:05.195961 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:58:05.195962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:58:05.196128 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:58:05.196175 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:05.196176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.197337 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.197338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:05.198122 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:58:05.198125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:58:05.198642 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:58:05.198643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:58:05.198727 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:05.199104 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:58:05.199108 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:58:05.200808 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:58:05.202672 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:05.202673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:05.202673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:05.202903 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:05.203115 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:58:05.203116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:58:05.203116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:58:05.203174 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:58:05.203441 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:58:05.203442 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:58:05.203485 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 22:58:05.203549 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 22:58:05.203642 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 22:58:05.205786 4658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802: Bootstrap starting.
I20260928 22:58:05.206755 4658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802: Neither blocks nor log segments found. Creating new log.
I20260928 22:58:05.207124 4658 log.cc:824] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802: Log is configured to *not* fsync() on all Append() calls
I20260928 22:58:05.207460 4739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a72a333765aa4cd9a006339fee99bffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } } }
I20260928 22:58:05.207566 4739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:58:05.207758 4777 mvcc.cc:205] Tried to move back new op lower bound from 7334446224166817792 to 7334446224105275392. Current Snapshot: MvccSnapshot[applied={T|T < 7334446224166817792}]
I20260928 22:58:05.207911 4753 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a72a333765aa4cd9a006339fee99bffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } } }
I20260928 22:58:05.208015 4753 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [sys.catalog]: This master's current role is: LEADER
I20260928 22:58:05.208141 4775 catalog_manager.cc:1347] Generated new cluster ID: e858c1cde2684b8a84758cc1a2749ec6
I20260928 22:58:05.208191 4775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:58:05.208294 4776 mvcc.cc:205] Tried to move back new op lower bound from 7334446224166817792 to 7334446224105275392. Current Snapshot: MvccSnapshot[applied={T|T < 7334446224166817792 or (T in {7334446224166817792})}]
I20260928 22:58:05.210619 4658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802: No bootstrap required, opened a new log
I20260928 22:58:05.212447 4658 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } }
I20260928 22:58:05.212579 4658 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad763a8b38024bc08eb1764247791802, State: Initialized, Role: FOLLOWER
I20260928 22:58:05.212644 4658 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } }
I20260928 22:58:05.212955 4658 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:58:05.214196 4632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a72a333765aa4cd9a006339fee99bffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ad763a8b38024bc08eb1764247791802"
I20260928 22:58:05.214300 4632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802 [term 0 FOLLOWER]: Advancing to term 1
W20260928 22:58:05.214803 4793 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:58:05.214911 4793 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:58:05.215448 4632 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72a333765aa4cd9a006339fee99bffa in term 1.
I20260928 22:58:05.215834 4631 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802 [term 1 FOLLOWER]: Refusing update from remote peer a72a333765aa4cd9a006339fee99bffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:58:05.219265 4772 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:58:05.212903 4782 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } } }
I20260928 22:58:05.220752 4782 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:58:05.220827 4782 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a72a333765aa4cd9a006339fee99bffa. Latest consensus state: current_term: 1 leader_uuid: "a72a333765aa4cd9a006339fee99bffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } } }
I20260928 22:58:05.220901 4782 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:58:05.220935 4782 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a72a333765aa4cd9a006339fee99bffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } } }
I20260928 22:58:05.220989 4782 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:58:05.233326 4796 mvcc.cc:205] Tried to move back new op lower bound from 7334446224166817792 to 7334446224105275392. Current Snapshot: MvccSnapshot[applied={T|T < 7334446224166817792}]
I20260928 22:58:05.256428 4772 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a72a333765aa4cd9a006339fee99bffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72a333765aa4cd9a006339fee99bffa" member_type: VOTER last_known_addr { host: "127.14.14.190" port: 44953 } } peers { permanent_uuid: "ad763a8b38024bc08eb1764247791802" member_type: VOTER last_known_addr { host: "127.14.14.189" port: 41309 } } peers { permanent_uuid: "6facde8029b5496d8fc6b8acd6e791b0" member_type: VOTER last_known_addr { host: "127.14.14.188" port: 41181 } } }
I20260928 22:58:05.256577 4772 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa [sys.catalog]: This master's current role is: LEADER
I20260928 22:58:05.258755 4775 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:58:05.259464 4775 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:58:05.266266 4775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a72a333765aa4cd9a006339fee99bffa: Generated new TSK 0
I20260928 22:58:05.266450 4775 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:58:05.294436 4586 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:05.292999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.14.14.190:44953)
0928 22:58:05.293054 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.293056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.293754 (+ 698us) server_negotiation.cc:407] Connection header received
0928 22:58:05.293865 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.293866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.293879 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.293910 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.294365 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46618: BlockingRecv error: recv got EOF from 127.0.0.1:46618 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:58:05.295209 4662 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:05.293414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.14.14.189:41309)
0928 22:58:05.293456 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.293458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.294696 (+ 1238us) server_negotiation.cc:407] Connection header received
0928 22:58:05.294794 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.294796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.294807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.294836 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.295150 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41054: BlockingRecv error: recv got EOF from 127.0.0.1:41054 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:58:05.296360 4736 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:05.295566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.14.14.188:41181)
0928 22:58:05.295626 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.295628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.295748 (+ 120us) server_negotiation.cc:407] Connection header received
0928 22:58:05.295841 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.295842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.295852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.295880 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.296297 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33554: BlockingRecv error: recv got EOF from 127.0.0.1:33554 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 22:58:05.298949 4662 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:05.298090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.14.14.189:41309)
0928 22:58:05.298124 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.298126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.298321 (+ 195us) server_negotiation.cc:407] Connection header received
0928 22:58:05.298427 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.298427 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.298464 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.298494 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.298897 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41068: BlockingRecv error: recv got EOF from 127.0.0.1:41068 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 22:58:05.302193 4662 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:05.301280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.14.14.189:41309)
0928 22:58:05.301314 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.301316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.301652 (+ 336us) server_negotiation.cc:407] Connection header received
0928 22:58:05.301762 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.301763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.301774 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.301803 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.302146 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41082: BlockingRecv error: recv got EOF from 127.0.0.1:41082 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 22:58:05.306113 4662 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:05.305270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41096 (local address 127.14.14.189:41309)
0928 22:58:05.305324 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.305326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.305567 (+ 241us) server_negotiation.cc:407] Connection header received
0928 22:58:05.305715 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.305715 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.305726 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.305754 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.306068 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41096: BlockingRecv error: recv got EOF from 127.0.0.1:41096 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 22:58:05.312114 4662 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:05.311237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.14.14.189:41309)
0928 22:58:05.311274 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.311276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.311534 (+ 258us) server_negotiation.cc:407] Connection header received
0928 22:58:05.311662 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.311663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.311675 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.311705 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.312056 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41100: BlockingRecv error: recv got EOF from 127.0.0.1:41100 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 22:58:05.322422 4662 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:05.321398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.14.14.189:41309)
0928 22:58:05.321442 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.321444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.321803 (+ 359us) server_negotiation.cc:407] Connection header received
0928 22:58:05.321912 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.321913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.321925 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.321954 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.322363 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41110: BlockingRecv error: recv got EOF from 127.0.0.1:41110 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 22:58:05.340821 4662 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:05.339787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.14.14.189:41309)
0928 22:58:05.339831 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.339834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.340129 (+ 295us) server_negotiation.cc:407] Connection header received
0928 22:58:05.340232 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.340233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.340244 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.340293 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.340759 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41120: BlockingRecv error: recv got EOF from 127.0.0.1:41120 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:58:05.375455 4662 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:05.374487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41126 (local address 127.14.14.189:41309)
0928 22:58:05.374537 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.374539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.374830 (+ 291us) server_negotiation.cc:407] Connection header received
0928 22:58:05.374905 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.374906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.374919 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.374999 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.375394 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41126: BlockingRecv error: recv got EOF from 127.0.0.1:41126 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:58:05.441825 4662 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:05.440953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.14.14.189:41309)
0928 22:58:05.441040 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.441042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.441149 (+ 107us) server_negotiation.cc:407] Connection header received
0928 22:58:05.441258 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.441258 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.441271 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.441364 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.441763 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41136: BlockingRecv error: recv got EOF from 127.0.0.1:41136 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 22:58:05.572057 4662 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:05.571088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.14.14.189:41309)
0928 22:58:05.571166 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.571168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.571293 (+ 125us) server_negotiation.cc:407] Connection header received
0928 22:58:05.571416 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.571417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.571451 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.571508 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.571976 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41138: BlockingRecv error: recv got EOF from 127.0.0.1:41138 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 22:58:05.829902 4662 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:58:05.829231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.14.14.189:41309)
0928 22:58:05.829296 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:05.829298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:05.829339 (+ 41us) server_negotiation.cc:407] Connection header received
0928 22:58:05.829366 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:05.829367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:05.829377 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:05.829424 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:58:05.829825 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41150: BlockingRecv error: recv got EOF from 127.0.0.1:41150 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:58:06.166170 4750 catalog_manager.cc:1259] Loaded cluster ID: e858c1cde2684b8a84758cc1a2749ec6
I20260928 22:58:06.166370 4750 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0: loading cluster ID for follower catalog manager: success
I20260928 22:58:06.167181 4750 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0: acquiring CA information for follower catalog manager: success
I20260928 22:58:06.167495 4750 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6facde8029b5496d8fc6b8acd6e791b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:58:06.215397 4793 catalog_manager.cc:1259] Loaded cluster ID: e858c1cde2684b8a84758cc1a2749ec6
I20260928 22:58:06.215474 4793 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802: loading cluster ID for follower catalog manager: success
I20260928 22:58:06.216334 4793 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802: acquiring CA information for follower catalog manager: success
I20260928 22:58:06.216667 4793 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ad763a8b38024bc08eb1764247791802: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:58:06.345624 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:06.343089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.14.14.189:41309)
0928 22:58:06.343239 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:06.343241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:06.343294 (+ 53us) server_negotiation.cc:407] Connection header received
0928 22:58:06.343361 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:06.343361 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:06.343414 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:06.343478 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:58:06.343619 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:06.344110 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:06.344913 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:06.345098 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:06.345148 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:06.345295 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:58:06.345403 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:58:06.345461 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:06.345493 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 22:58:06.345511 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260928 22:58:06.350467 4870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:06.347683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.14.14.188:41181)
0928 22:58:06.347861 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:06.347863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:06.347872 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:58:06.347911 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:06.347911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:06.347947 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:06.348027 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:58:06.348221 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:06.348777 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:06.349501 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:06.349837 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:06.349894 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:06.350107 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:58:06.350228 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:58:06.350291 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:06.350323 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 22:58:06.350345 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":98,"threads_started":1}
I20260928 22:58:06.355490 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:06.352298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.14.14.190:44953)
0928 22:58:06.352390 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:06.352393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:06.352401 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:58:06.352453 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:06.352454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:06.352509 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:06.352873 (+ 364us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:58:06.352945 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:06.353737 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:06.354512 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:06.354757 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:06.354827 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:06.354980 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:58:06.355083 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:58:06.355332 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:06.355350 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 22:58:06.355374 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":47,"threads_started":1}
I20260928 22:58:06.355559 4870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:06.352430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.14.14.188:41181)
0928 22:58:06.352495 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:06.352497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:06.352615 (+ 118us) server_negotiation.cc:407] Connection header received
0928 22:58:06.352699 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:06.352700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:06.352712 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:06.352741 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:58:06.352948 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:06.353714 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:06.354766 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:06.355056 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:06.355136 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:06.355275 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:58:06.355308 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:58:06.355364 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:06.355467 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 22:58:06.355491 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:58:06.356189 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:58:06.352228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.14.14.189:41309)
0928 22:58:06.352390 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 22:58:06.352393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:58:06.352519 (+ 126us) server_negotiation.cc:407] Connection header received
0928 22:58:06.352756 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:58:06.352758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:58:06.352769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:58:06.352827 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:58:06.353727 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:06.354474 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:06.355439 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:58:06.355712 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:58:06.355775 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:58:06.355945 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:58:06.355980 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:58:06.356032 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:58:06.356092 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 22:58:06.356113 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260928 22:58:06.357488 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 4515
I20260928 22:58:06.362864 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 4590
I20260928 22:58:06.367981 14394 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdZKSl/build/release/bin/kudu with pid 4665
2026-09-28T22:58:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2857 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5634 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (270119 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST